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Overview 

The Massachusetts Department of Elementary and Secondary Education (Department) created the 
grades 1-12 Early Warning Indicator System (EWIS) in response to district interest in the Early Warning 
Indicator Index (EWII) that the Department previously created for rising grade 9 students. Districts 
shared that the EWII data were helpful, but also requested early indicator data at earlier grade levels 
and throughout high school. The new EWIS builds on the strengths and lessons learned from the EWII to 
provide early indicator data for grades 1-12. 

The Department worked with American Institutes for Research (AIR) to develop the new risk models for 
the EWIS. AIR has extensive experience with developing early warning systems and supporting their use 
at the state and local levels. AIR conducted an extensive literature review of the research on indicators 
for early warning systems. AIR then identified and tested possible indicators for the risk models based 
on those recognized in the research and data that are collected and available from the Department's 
data system. Because of limitations in the availability of data for children from birth through pre- 
kindergarten, the students from kindergarten through twelfth grade were the focus of EWIS statistical 
model testing. Massachusetts' longitudinal data system allowed estimated probabilities of being at risk 
on the predefined outcome measures for students based on previous school years. The model for each 
grade level was tested and determined separately. While there are some common indicators across age 
groupings and grade levels, the models do vary by grade level. A team from ESE worked closely with AIR 
in determining the recommended models for each grade level and an agency-wide EWIS advisory group 
reviewed research findings and discussed key decisions. 

To develop the early elementary risk model, we used a multilevel modeling framework to control for the 
clustering of students within schools and obtain correct robust standard errors (Raudenbush & Bryk, 
2002). To develop the late elementary, middle and high school risk models, we used a logistic regression 
modeling framework * 1 . The model allows users to identify students who are at risk of missing key 
educational benchmarks (a.k.a. outcome variables) within the first through twelfth grade educational 
trajectory. The outcome variables by which students risk is tested took into consideration the degree to 
which the outcome variable is age and developmentally appropriate (e.g., achieving a score that is 
proficient or higher on the third grade English Language Arts in Massachusetts Comprehensive 
Assessment System). 

The following research questions guided the development of the EWIS statistical model that helps 
identify risk levels for individual students: What are the indicators (or combination of indicators) that 
predict whether are at risk of missing key educational benchmarks in Massachusetts that are above and 
beyond student demographic characteristics, based on predefined student clusters and appropriate 
outcome variables? 

Identification of at-risk students through the risk model developed for each age group served as the 
foundation of the EWIS, which aims to support practitioners in schools and districts to identify 
children/students who may be at risk. With this relevant and timely information, teachers, educators, 


1 HGLM models were not be able to used in the middle school and high school age groups since development of 
these age groups relied on a sample of district student course data, and therefore could not estimate the 
statewide school random effects for prediction. The late elementary model was updated to use more recent 
assessment data and, due to time constraints, the logistical regression model was employed. As state data 
become available for the middle and high school models, ESE will consider the feasibility of HGLM for EWIS model 
development. ESE will also consider whether to employ HGLM with late elementary models. 
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and program staff will be able to intervene early and provide students with the targeted support. The 
EWIS identification of at-risk students is designed to provide an end of year indicator, which is 
cumulative for an academic year of school and identifies students with a risk designation to inform 
supports in the next school year. 

Age Groups and Outcome Measures 

Students are grouped by grade levels and related academic goals were identified that are 
developmental^ appropriate, based on available state data, and meaningful to and actionable for adult 
educators who work with the students in each grade grouping. Each academic goal is relevant to the 
specific age grouping, and also ultimately connected with the last academic goal in the model: high 
school graduation. 

For example, the early elementary age group encompasses grades one through three, and assesses risk 
based on the academic goal of achieving a score of proficient or higher on the third grade ELA MCAS, a 
proxy for reading by the end of third grade, a developmentally appropriate benchmark for children in 
the early grades. Reading by the end of the third grade is also associated with the final academic goal in 
the model of high school graduation. Exhibit 1.1 provides an overview of the age groups and outcome 
variables for the risk model. 


Exhibit 1.1 Overview of Massachusetts EWIS age groups and outcome variables 


Age Groups 

Grade Levels 

Academic Goals 

(expected student outcomes for each age group) 

Early 

Elementary 

Grades 1-3 

Proficient or advanced on 3rd grade ELA MCAS 

Late 

Elementary 

Grades 4-6 

Proficient or advanced on 6 th grade ELA and Mathematics 
MCAS 

Middle 

Grades 

Grades 7-9 

Passing grades on all 9 th grade courses 

High 

School 

Grades 10-12 

High school graduation 


Risk Indicators 

The risk indicators tested in the Massachusetts' risk model are comprised of indicators that have been 
identified in research, as well as data elements that are collected and available from the ESE data 
system. Many of the indicators are dependent on the availability of ESE student level data over a 
number of years. * 2 Since 2002 ESE has collected extensive individual student information through 
Student Information Management System (SIMS). SIMS data provided information on student 
demographics, enrollment, attendance, and suspensions, with a unique statewide identification code (a 
State-Assigned Student Identifier, SASID). Recently, ESE has begun collecting course taking and course 
performance data at the middle and high school levels. Although these data have not been collected for 
enough years (at least six years) to use statewide data for the development of the EWIS model, a sample 
of eight urban and suburban districts provided longitudinal course taking and course performance data 
so that these variables could be included into the middle and high school models. In turn, these data 


2 At the middle and high school grades a sample of districts provided student course taking and course 
performance data to develop the EWIS risk model. The sample for the middle and high school model development 
is therefore much smaller. 
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were linked to SIMS data. By linking SIMS data across years, this study was able to identify whether a 
student moved school during a school year and whether a student was retained in grade. 

Risk Levels 

There are three risk levels in the EWIS: low, moderate, and high risk. The risk levels relate to a student's 
predicted likelihood for reaching a key academic goal if the student remains on the path they are 
currently on (absent interventions). In other words, the risk level indicates whether the student is 
currently "on track" to reach the upcoming academic goal. A student that is "low risk" is predicted to be 
likely to meet the academic goal. The risk levels are determined using data from the previous school 
year. The risk levels are determined on an individual student basis and are not based on a student's 
relative likelihood for reaching an academic goal when compared with other students. As a result there 
are no set amounts of students in each risk level. For example, it is possible to have all students in a 
school in the low risk category. 


Exhibit 1.2 Massachusetts Early Warning Indicator System: Risk Levels 


Indicates that, based on data from last school year, the student is... 

Low risk 

likely to reach the upcoming academic goal 

Moderate risk 

moderately at risk for not reaching the upcoming academic goal 

High risk 

at risk for not reaching the upcoming academic goal 


Validating the Risk Models 

Once the models were finalized, the risk model for each grade level was validated using a second cohort 
of student data (e.g., the 2008-09 third grade cohort to the 2009-10). The intent of this step is to 
examine the extent to which the finalized risk model, developed using the original cohort data, correctly 
identifies at risk students in the validation cohort in terms of those who met or exceeded the risk 
thresholds (low, moderate, high) of the predefined outcome measure. 

The following procedure was followed to make this determination. First, regression coefficients were 
compared in terms of the direction of the estimated coefficient and its statistical significance in each 
individual variable by running the same model for the validation cohort data. Second, the accuracy of 
prediction was examined by applying the equation of the already developed 'Final' EWIS risk model to 
the validation cohort data. Comparisons were made between the original cohort data and validation 
data to see whether the validation cohort showed the same level of prediction accuracy in the 
proportion of students who were classified as at risk and actually did not meet or exceeded the risk 
threshold of the outcome variable. 

Final Risk Model 

Exhibit 1.3 provides an overview of the indicators that are included in the models based on the testing 
and validation of the Massachusetts Early Warning Indicator System Risk Model for the early 
elementary, late elementary, middle school and high school age groups. The list of indicators is 
representative of some of those that were tested. In grades where the tested indicators are marked 
with an "x," these indicators were found to add to the predictive probability of the model and are 
included in the model. 
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Exhibit 1.3 Overview of the final EWIS model, by grade level 



I 

Grade Level 


Age Group 

Outcome Variable 

Early Elementary 

Proficient or 

Late Elementary 

Proficient or Advanced 

Middle School 

High School 


Advanced on 3rd 

on 6th Grade ELA & 

Pass all Grade 9 

Graduate from HS in 


Grade ELA MCAS 

Math MCAS 

Courses 

4 years 

Indicators Included in 

Risk Model 

^ st 2 nd 3 rd 

4 th 5 th 6 th 

3 

00 

3 

3 

2Qth <| th >| th 


Attendance rate 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

School move 













(in single year) 













Number of in-school and 
out-of-school suspensions 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

MEPA Levels 




X 

X 

X 

X 

X 

X 

X 

X 


ELA MCAS 




X 

X 

X 

X 

X 

X 




Math MCAS 




X 

X 

X 

X 

X 

X 

X* 



Retained 


X 

X 

X 

X 

X 

X 


X 

X 



Low income 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 


Special education 













level of need 












X 

ELL status 

X 

X 

X 










Gender 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Urban residence 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Overage for grade 

X 

X 

X 

X 

X 

X 




X 

X 

X 

School wide Title 1 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Targeted Title 1 

X 

X 

X 

X 

X 

X 







Math course performance 







X 

X 


X 

X 

X 

ELA course performance 







X 


X 

X 

X 

X 

Science 

course performance 







X 


X 

X 

X 

X 

Social studies course 
performance 








X 


X 

X 

X 

Non-core 

course performance 







X 

X 

X 

X 

X 

X 


Notes: 


• In grades where the tested indicators are marked with an "x," these indicators were found to add to the predictive 
probability of the model, typically at an alpha level of .10. We chose a less conservative critical alpha level, because 
overidentification was preferred over underidentification in order to reduce the risk of excluding students in need of 
support or intervention, and because the risk models of middle and high school age groups were based on district data 
instead of state-wide data. Additional consideration was also given to consistency of models, especially in the middle and 
high school age groupings when dealing with smaller sample sizes. 


• Mobility was initially tested for middle and high school age groupings, but due to use of course performance data from a 
subset of districts, the variable was excluded. A large proportion of students who moved schools within the school year 
ended up lacking sufficient course performance information and/or not being part of the outcome sample (by ninth grade 
they were not enrolled in a school that was taking part in the data pilot). 
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• Due to small sample in individual MEPA levels in middle and highschool, final model aggregates MEPA levels beginner to 
intermediate as a single indicator, leaving transiting to regular classes and non-MEPA as 0 for this variable. The benefit of 
this strategy is that this indicator fits in the EWIS models with the current MEPA levels having 5 categories. Thus, the 
binary indicator of MEPA levels was used for many of the EWIS models. 

• The 10 th grade model (built using data from 9 th grade students) uses the MCAS score from 8 th grade since 9 th grade is not a 
tested MCAS grade. ELA MCAS results were not available for use in 10th grade model due to available years of data. 8 th 
grade ELA MCAS was first administered in 2006 and so could not be used in developing the model since data was not 
available for validation. This variable will be tested for inclusion in future years. 

• Retention variable was not used as an indicator in high school age grouping, because the variable was directly related to 
the outcome benchmark in high schools, i.e., on-time graduation. 

• Special education variable has 4 categories based on levels of need of special education: 1) Low- less than 2 hours, 2) Low - 
2 or more hours, 3) Moderate, and 4) High. Each indicators denoting individual level of need were tested. However, due to 
data limitations with small sample sizes in middle and high school age grouping, the directions and magnitudes of the 
coefficients appeared inappropriate. Thus, we ended up using a binary indicator covering low to high levels of need (2 
hours or more) in the middle and high school age group. We plan retesting individual indicators representing each level of 
need in special education when state-wide data are available. 

• Overage for early elementary, late elementary and middle school is defined as one year older than the expected age for 
the grade level. For the high school, students two or more years older than expected grade level are considered overage. 

• Due to data limitations with smaller sample size with middle and high school age groupings. Targeted Title I was 
miniminally represented, so only school wide Title I is in middle and high school age grouping models. 

• Variables indicating whether a student did not enroll in or miss a certain subject ('flagged') were not tested in middle 
schools, because the numbers of students in falling in this category were too small (less than 2%). 
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Early Elementary Age Group (First through Third Grade) 

The Early Elementary Age Group encompasses first through third grade, using data from students during 
their kindergarten, first and second grade year. Within the age group indicators of risk were tested at 
each grade level based on the outcome variable of scoring proficient or higher on the Third Grade 
English Language Arts (ELA) of Massachusetts Comprehensive Assessment System (MCAS). The 
outcome variable is chosen as a proxy for reading by the end of third grade benchmark. 

Potential Indicators 

In the Early Elementary Age Group, the indicators tested included behavioral, demographic and other 
variables. Behavioral indicators are mutable and considered manifestations of student behavior (e.g., 
attendance, suspensions). Demographic indicators are tied to who the child is, and are not necessarily 
based on a student's behavior (although some of these, such as low income household, may change 
over time). Last, other individual student variables are focused on characteristics related to the type of 
services the student receives. Exhibit Early Elementary.l provides an overview and definition of the 
indicators by variable 3 . 


Exhibit Early Elementary.l. Indicator Definitions, by Type 


Type 

Indicator 

Definition 

Corresponding 

Data Source 

Outcome variables 



Third Grade 
English 

Language Arts 
MCAS 

Binary variable: 1= Proficient or above proficient; 0=Warning or needs 
improvement 

Indicates students who achieve a proficient (or higher) or below 
proficient score on the Third Grade ELA MCAS 

MCAS 2010 data 

variable name: 

EPERF2 

Behavioral variables 



Attendance 

Continuous variable: Attendance rate, end of year- number of days in 
attendance over the number of days in membership 

SIMS DOE045 

SIMS DOE046 


Suspension 

Continuous variable: Suspensions, end of year - number of days in 
school suspension plus number of days out of school suspension 

SIMS DOE017 

SIMS DOE018 


Retention 4 

Binary variable: Based on whether child is listed as same grade 
between two consecutive years l=Retained; 0=Not retained 

SIMS DOE016 


Mobility 

Binary variable: l=School code changes from beginning of school year 
to end of school year; 0= School code is the same at beginning and end 
of school year 

SIMS 8 digit school 
identifier 

Demographic variables 



Gender 

Binary variable: l=Female; 0=Male 

SIMS DOE009 


Low income 

household - 

Free lunch 

Binary variable: l=Free lunch eligible; 0= not eligible 

SIMS DOE019 


Low income 

household - 
Reduced price 
lunch 

Binary variable: l=Reduced lunch recipient; 0= Not eligible for reduced 
price lunch 

SIMS DOE019 


Overage for 
grade 

Binary variable: l=Age of child is equal to or greater than one year 
than expected grade level age as of September 1 in a given calendar 

SIMS DOE006 


3 The table includes all variables tested in the Early Elementary Age Group, but there may be variation in which of 
these were tested in individual grades. For example, 'Kindergarten, full day' was only tested for the first grade 
model. 

4 Retention is defined from fall to fall. 
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Type 

Indicator 

Definition 

Corresponding 

Data Source 



year; 0= Age of child is less than one year than expected grade level 
age (e.g. a student who is 8 as of September 1 st of their second grade 
year is overage). 







Immigration 

Status 

Binary variable: 1= Student is an immigrant under the federal 
definition; 0=Student is not an immigrant 

SIMS DOE022 


Urban 

residence 

Binary variable: l=Student lives in an urban area b ; 0= Student does not 
live in one of the specified urban areas 

SIMS DOE014 


ELL program 

Binary variable: 1= sheltered English Immersion (SEI) or 2-way 
bilingual or other; 

0 = opt out, no program 

SIMS DOE026 


Special 

Education - 

Level of Need 

Special Education - Multiple indicators 

• Dummy variable: Low level of need (less than 2 hours) is 
equal to 1; otherwise 0. 

• Dummy variable: Low level of need (2 or more hours) is 
equal to 1; otherwise 0. 

• Dummy variable: Moderate level of need is equal to 1; 
otherwise 0. 

• Dummy variable: High level of need is equal to 1; otherwise 

0 . 

SIMS DOE038 

Other individual student variables 



Title 1 

participation 

Binary variables: 

• Targeted Title 1, Binary variable: 1= Any type of targeted 

Title 1 participation; 0= Not included in targeted Title I 6 

• School -wide Title 1, Binary variable: 1= School-wide Title 1; 

0= Not school-wide Title 1 

SIMS DOE020 


Kindergarten - 
Full day 

Binary variable: 1 = either full-time kindergarten or full-time 
kindergarten, tuitioned; otherwise 0. 

SIMS DOE016 


Analysis Methods and Strategies 

To identify the model that most accurately predicted risk of students who do not achieve proficiency on 
third grade ELA MCAS, multiple analyses were conducted. For prediction of the third grade ELA MCAS 
proficiency, a separate analysis was conducted in each grade to predict a risk level for students as they 
entered the next year: first grade (using students' kindergarten data), second grade (using students' 
grade 1 data), and third grade (using students' grade 2 data). 

Developing the Risk Model by Grade 

For the data analysis, we focused on the 2009-10 third grade cohort of students with valid ELA MCAS 
performance scores. SIMS data in 2006-07 through 2009-10 were analyzed to identify the predictive 
indicators in each grade (see Exhibit Early Elementary.2). 


5 Specified urban areas: Boston, Brockton, Cambridge, Chelsea, Chicopee, Everett, Fall River, Fitchburg, 
Framingham, Haverhill, Flolyoke, Lawrence, Leominster, Lowell, Lynn, Malden, New Bedford, Pittsfield, Quincy, 
Revere, Somerville, Springfield, Taunton, Worcester. These were the urban districts during the years tested. 

6 There is only one possible outcome per student for the Title I variable, so if they are elected as school-wide Title I 
they cannot be considered targeted and vice versa according to the data. 
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Exhibit EarlyElementary.2. Numbers of students and schools by data source 

3 rd grade Proficiency in ELA 
MCAS 


Source Data 

Below 

Threshold 

Proficient or 

Above 

# Students 

# Schools 

3rd grade in 2009-10 

26,234 (37%) 

44,433 (63%) 

70,667 

1,105 

Kindergarten in 2006-07 
(used to create 1 st grade model) 

20,813 (35%) 

38,655 (65%) 

59,468* 

1,094 

Grade 1 in 2007-08 

(used to create 2 nd grade model) 

23,487 (36%) 

41,812 (64%) 

65,299 

1,136 

Grade 2 in 2008-09 

25,062 (37%) 

43,212 (63%) 

68,274 

1,107 


(used to create 3 grade model) 

* Denotes the number of kindergarten students in 2006-07 who tested third grade MCAS assessment in 2009-10 SIMS data, and 
11,199 third grade students out of 70,667 (16.4%) in 2009-10 have missing information in 2006-07 SIMS kindergarten data. 

The following strategies were employed in each grade level analysis. 

• First, in order to build an efficient and accurate model for the EWIS, we examined a number of 
behavioral, demographic, and other individual student variables that may be considered in the 
resulting risk model. This analysis relied on simple logistic regressions for each individual 
indicator. The individual indicator analyses allowed us to evaluate the statistical significance 
and coefficient for each indicator. This analysis was used to inform the construction of the risk 
models tested. 

• Then, based on the results of the simple logistic regression models, a series of analysis were 
conducted - 

o Student behavioral variables only 7 ; 

o Demographic variables along with the behavioral variables from the previous model; 
o Demographic variables, behavioral variables, and individual student variables including 
the availability of school wide and targeted Title I; 
o Multi-level logistic regression 8 to account for the clustering of students within schools, 
and allowed the level-1 intercept to be random; and 
o Multi-level logistic regression models with random intercept and slope were also tested, 
which enabled to examine whether the associations between level-1 (student) 
indicators and the outcome measure (not achieving proficiency on third grade ELA 
MCAS) vary across schools. 9 


7 The analysis began with behavior variables because we wanted to identify variables that are mutable, as opposed 
to demographic variables that are related to who a student is, rather than the behaviors he or she exhibits. 

8 The SAS GLIMMIX procedure was used for data analysis. 

9 Attention was paid to whether and to what extent a random slope model actually helps substantially improve 
prediction of identifying at-risk students. Because the ultimate goal of the EWIS is to apply the fitted models to 
another student cohort data and to obtain the predictive risk levels for individual students in the upcoming year, 
development of viable and robust statistical models is important. 
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First Grade: Analysis Results and Predicted Risk Levels 

For the first grade model, models were tested to: 1) identify individual indicators of risk and 2) identify 
the risk model that is most predictive of whether a rising first grade student is at risk of not meeting the 
academic goal of achieving a score that is proficient or higher on the third ELA MCAS (Exhibit Gradel.l). 


Exh 


bit Gradel.l Overview of First Grade Risk Indicators 

Grade: 

First Grade (using data from Kindergarteners) 

Age Grouping: 

Early elementary (1 st -3 rd grade) 

Risk Indicators Tested: 

Behavioral variables 


• 

Suspensions, fall 


• 

Suspensions, end of year 


• 

Attendance rate, fall 


• 

Attendance rate, end of year 


• 

Mobility (more than one school within the school year) 


Demographic variables 


• 

Low income household- Free lunch 


• 

Low income household- Reduced price lunch 


• 

Special education level variables (4 total) 


• 

ELL status 


• 

Immigration status 


• 

Gender 


• 

Urban residence 


• 

Overage for grade (age 6 or older by sept 1 st of Kindergarten) 


Other individual student variables 


• 

Kindergarten, full day 


• 

School wide Title 1 


• 

Targeted Title 1 

Academic Goal/ 

Proficient or higher on the third grade English language arts MCAS (proxy 

Outcome Variable 10 : 

for reading by third grade) 


NOTE: A total of 59,468 observations included this outcome variable for the final model. Approximately 65 percent were 
characterized as proficient or above, and the remaining 35 percent were less than proficient. 


First Grade: Simple Logistics - Analysis of Individual Indicators 

We first examined a number of behavioral, demographic, and other indicators tied to individual students 
that may be considered in the resulting risk model. This analysis relied on simple logistic regressions for 
each individual indicator. The single indicator analyses allowed us to evaluate the statistical significance 
and coefficient for each indicator (Exhibit Gradel.2). This analysis was used to inform the construction 
of the risk models tested (Exhibit Gradel.3). 


10 For running the statistical regression models, the outcome variable was recoded to predict the risk/likelihood of not being 
proficient or higher on the third grade ELA MCAS. 
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Exhibit Gradel.2. Simple Logistic Regression Overview, Grade 1 


Simple Logistic regression: Individual indicators (predictor) 


Variable 

Estimate 

S.E. 

Pr > ChiSq 

R-Square 

N 

Demographic variables (Yes/No) 






Low income household- Free lunch 

1.34 

0.02 

<.0001 

0.0726 

59,468 

Low income household- Reduced price lunch 

0.88 

0.04 

<.0001 

Special education 




0.0552 

59,468 

Low level of need (less than 2 hours) 

0.55 

0.04 

<.0001 

Low level of need (2 or more hours) 

1.41 

0.06 

<.0001 

Moderate level of need 

1.55 

0.04 

<.0001 

High level of need 

2.72 

0.09 

<.0001 

ELL status 

1.14 

0.03 

<.0001 

0.0227 

59,468 

Immigration statust 

0.44 

0.06 

<.0001 

0.0008 

59,468 

Sex: Female 

-0.39 

0.02 

<.0001 

0.0084 

59,468 

Overage for grade 

0.60 

0.03 

<.0001 

0.0049 

59,468 

Urban residence 

0.97 

0.02 

<.0001 

0.0464 

59,468 

Suspension 






Suspensions, fallt 

0.79 

0.25 

0.0014 

0.0002 

58,625 

Suspensions, end of year 

0.65 

0.07 

<.0001 

0.0023 

59,468 

Attendance 






Attendance rate, fallt 

-1.19 

0.10 

<.0001 

0.0025 

58,625 

Attendance rate, end of year 

-4.28 

0.17 

<.0001 

0.0120 

59,468 

Mobility - Changed schools during school year 
(Yes/No) 

0.80 

0.04 

<.0001 

0.0069 

59,468 

Full time Kindergarten (Yes/No) + 

0.58 

0.02 

<.0001 

0.0164 

59,468 

Title 1 participation (Yes/No) 






School-wide 

1.04 

0.02 

<.0001 

0.0515 

59,468 

Targeted 

1.16 

0.06 

<.0001 


Exhibit Reads: students overage for grade are 0.6 higher in the log-odds of not being proficient in ELA MCAS than 
others. 

tindicator was removed from final analyses, because either the direction of the coefficient of the variable was 
changed, or it was not statistically significant at an alpha level of .05 and the estimated coefficient is nearly zero. 

First Grade: Risk Models Overview and Final Model 

Several risk models were tested and additional predictive probability of the model was balanced with 
model complexity. Our findings from the examination are that the random intercept model is predictive 
and more complex models (e.g., random intercept and slope model 11 ) did not much add to the 


11 The percentage of students who were classified as "high risk" and actually did not meet the proficiency level on 
the third grade ELA MCAS was two percent higher in the random intercept and slope model than in the random 
intercept model. The difference appears trivial enough to select the more parsimonious model, i.e., a random 
intercept model. Furthermore, when the comparison was made for another age group, the difference was almost 
null (0.2 percent higher in a random intercept and slope model than in a random intercept model). Thus, the 


10 | Page 


predictive probability of the model and introduced unnecessary complexity to the risk model. Exhibit 
Gradel.3 provides an overview of four of these models. Model 4 is was the chosen First Grade model. 


Exhibit Gradel.3. Overview of Findings by Model, Grade 1 



Logistic Regression Analysis 

Random 

Intercept Model 12 

Variable 

Model 1 

Model 2 

Model 3 

Final model 

Model 4 

Behavioral variables 





Attendance, fallt 

-0.17 




Attendance, end of year 

-3 

_2 47 * ** 

-1.40*** 

-1.13*** 

Suspensions, fallt 

-0.24 




Suspensions, end of year 

0.58*** 

0.20*** 

0.20*** 

0.18*** 

Mobility, Changed schools by the end of year 

0.70*** 

0 35 ** * 

0 35*** 

0.33*** 

Demographic variables 





Low income household- Free lunch 


0 95*** 

0 90 *** 

0.80*** 

Low income household- Reduced price lunch 


0 59*** 

0.54*** 

0.47*** 

Special education status 





Low level of need (< 2 hours) 


0.63*** 

0.62*** 

0.66*** 

Low level of need (>2 hours) 


2 22 *** 

2 22 *** 

1.25*** 

Moderate level of need 


1.58*** 

1.58*** 

1.63*** 

High level of need 


2 ^^*** 

2.56*** 

2.61*** 

ELL status 


0.69*** 

0.63*** 

0.62*** 

Immigration status! 


-0.30*** 



Gender (Female) 


-0.28*** 

-0.28*** 

-0.28*** 

Urban residence 


0.41*** 

0.26*** 

0.33*** 

Overage for grade 


0 19*** 

0.20*** 

0.22*** 

Other Indicators 





Kindergarten - Full day + 



-0.03 


School wide Title 1 



q 27 ** * 

0.30*** 

Targeted Title 1 



0 91*** 

1.02*** 

r A 2 

0.0166 

0.1394 

0.1424 

n/a 

Variance of school-level intercept (standard error) 




0.2047(0.0144) 


Exhibit Reads: students that are overage for grade are 0.19 higher in the log-odds of not being proficient in ELA MCAS than 
others, holding constant other variables in Model 2. 

tlndicator was removed from final analyses, because either the direction of the coefficient of the variable was changed, or it 
was not statistically significant at an alpha level of .05 and the estimated coefficient is nearly zero. 

* Significant at 10%, **Significant at 5%, ***Significant at 1% 


random intercept model was determined the preferred model and applied to all early elementary age group 
analyses. 

12 Continuous variables (attendance and suspensions) were grand-mean centered, and all binary variables were not 
centered. 
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Exhibit Gradel.4 provides the summary statistics for the final model (Model 4). The estimates in column 
2 denote the expected difference in the log-odds of not being proficient in third grade ELA MCAS, 
holding constant other variables in the model. For example, students in low income households who 
are receiving free lunch are expected to score 0.80 points higher than other students in the log-odds of 
not being proficient in ELA MCAS, holding other variables constant. Additionally, based on the result of 
the statistically significant random intercept at the school level, we can infer that the likelihood of not 
being proficient in third grade ELA MCAS also depends on school membership, holding constant student- 
level characteristics. With the exception of attendance and gender variables, all other variables are 
statistically positively associated with the recoded outcome variable (not being proficient on the third 
grade ELA MCAS). 

Exhibit Gradel.4. Final Model - Behavioral Variables, Demographic Variables, and Other Indicators - 
Random Intercept Model, Summary Statistics 


Variable 

Estimate 

S.E. 

Pr > |t| 

Behavioral variables 

Attendance rate, end of year 

-1.13 

0.16 

<.0001 

Suspensions, end of year 

0.18 

0.06 

0.0038 

Mobility - Changed schools during school year 

0.33 

0.04 

<.0001 

Demographic variables 

Low income household- Free lunch 

0.80 

0.03 

<.0001 

Low income household- Reduced price lunch 

0.47 

0.04 

<.0001 

Special Education 

Low level of need (less than 2 hours) 

0.66 

0.05 

<.0001 

Low level of need (2 or more hours) 

1.25 

0.07 

<.0001 

Moderate level of need 

1.63 

0.05 

<.0001 

High level of need 

2.61 

0.09 

<.0001 

ELL Status 

0.62 

0.04 

<.0001 

Sex: Female 

-0.28 

0.02 

<.0001 

Urban residence 

0.33 

0.06 

<.0001 

Overage for grade 

0.22 

0.04 

<.0001 

Other variables 

School wide Title 1 

0.30 

0.06 

<.0001 

Target Title 1 

1.02 

0.07 

<.0001 


Variance of school-level intercept (standard error): 0.2047 (0.0144) 
Number of observations=59468 


First Grade: Illustration of Levels of Risk and MCAS Outcomes Using the Final Model 

The following box plot shows the distribution of test scores by increased risk, using the First Grade EWIS 
Model As the risk level increases (x axis), the proficiency level appears to decrease (y axis). The levels of 
risk are defined as follows: 

• Low Risk (approximately 75% or more of students meet the outcome variable): Intervals 1-3; 

• Moderate Risk (approximately half or more than half of the students meet the outcome 
variable) : Intervals 4-5; and 
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• High Risk (approximately a third or less of the students meet the outcome variable): Intervals 6- 
9. 13 

Exhibit Gradel.5. Box Plot Distribution, Grade 1 

First Grade Risk Based on Early Elementary Outcome: Proficient or Advanced on Grade 3 ELA MCAS 



123456789 

Increased risk level 


The statistics for the Final model's three levels of risk (low risk; moderate risk and high risk) are shown in 
Exhibits Gradel.6 and Gradel.7. In summary, approximately 81 percent of students who fall into the 
low risk category have met the outcome variable of proficient or higher on the third grade ELA MCAS 
(Exhibit Gradel.7). Of the students who are categorized in the moderate risk category, approximately 
60 percent of the students have met the outcome variable. Among the high risk students approximately 
32 percent met the outcome variable and 68 percent of the students scored below proficient on the 
third Grade ELA MCAS. 


In order to improve the prediction accuracy, a more stringent risk level threshold could have been employed; 
however, overidentification was preferred over underidentification in order to reduce the risk of excluding 
students in need of support or intervention. 
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Exhibit Gradel.6. Final Model - Risk Level Based on Box Plot Distribution, Grade 1 


Total numbers of students in sample by risk levels 



Exhibit Gradel.7. Final Model - Predictive Probability of Proficiency Based on Risk Level, Grade 1 


Predictive Probability of Proficiency Based on Risk Level 


Proficiency 


Risk Level 

Below 

Threshold 

Proficient 

or Above 

Total 

Low 

6,039 

25,921 

31,960 

19% 

81% 

100% 

Moderate 

5,714 

8,383 

14,097 

41% 

59% 

100% 


9,060 

4,351 

13,411 

High 

68% 

32% 

100% 


20,813 

38,655 

59,468 

Total 

35% 

65% 

100% 
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Second Grade: Analysis Results and Predicted Risk Levels 

For second grade, several risk models were tested to: 1) identify individual indicators of risk and 2) 
identify the risk model that is predictive of whether a rising second grader is at risk of not meeting the 
academic goal of achieving a score that is proficient or above on the third grade English language arts 
Massachusetts Comprehensive Assessment System (ELA MCAS) based on their information from first 
grade (Exhibit Grade2.1). 


Exh 


bit Grade2.1 Overview of Second Grade Risk Indicators 

Grade: 

2 (Using Data from grade 1 students) 

Age Grouping: 

Early elementary (1 st -3 rd grade) 

Risk Indicators Tested: 

Behavioral variables 


• 

Suspensions, end of year 


• 

Suspensions, fall semester 


• 

Attendance rate, end of year 


• 

Attendance rate, fall semester 


• 

Mobility (more than one school within the school year) 


• 

Retention 


Demographic variables 


• 

Low income household- Free lunch 


• 

Low income household- Reduced price lunch 


• 

Special education level variables (4 total) 


• 

ELL status 


• 

Immigration status 


• 

Gender 


• 

Urban residence 


• 

Overage for grade (age 7 or older by Sept 1 st of 1 st grade year) 


Other individual student variables 


• 

School wide Title 1 


• 

Targeted Title 1 

Academic Goal/ 

Proficient or higher on the third grade English language arts MCAS (proxy 

Outcome Variable 14 : 

for reading by third grade) 


NOTE: A total of 65,299 observations 
characterized as proficient or above. 


included this outcome variable for the Final model. Approximately 65 percent were 
and the remaining 35 percent were less than proficient. 


Second Grade: Simple Logistics - Analysis of Individual Indicators 

In order to build the most efficient and accurate model for the EWIS, we first examined a number of 
behavioral, demographic, and other indicators tied to individual students that may be considered in the 
resulting risk model. This analysis relied on simple logistic regressions for each individual indicator. The 
single indicator analyses allowed us to evaluate the statistical significance and coefficient for each 
indicator (Exhibit Grade2.2). This analysis was used to inform the construction of the risk models tested. 


14 For running the statistical regression models, the outcome variable was recoded to predict the risk/likelihood of not being 
proficient or higher on the third grade ELA MCAS. 
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Exhibit Grade2.2. Simple Logistic Regression Overview, Grade 2 


Simple Logistic regression: Individual indicators (predictor) 


Variable 

Estimate 

S.E. 

Pr > ChiSq 

R-Square 

N 

Demographic variables (Yes/No) 






Low income household- Free lunch 

1.39 

0.02 

<.0001 

0.0848 

65,299 

Low income household- Reduced price 
lunch 

0.91 

0.04 

<.0001 

Special education 






Low level of need (less than 2 hours) 

0.66 

0.04 

<.0001 

0.0769 

65,299 

Low level of need (2 or more hours) 

1.45 

0.05 

<.0001 

Moderate level of need 

1.75 

0.04 

<.0001 

High level of need 

2.86 

0.08 

<.0001 

ELL status 

1.31 

0.03 

<.0001 

0.0291 

65,299 

Immigration statust 

0.58 

0.06 

<.0001 

0.0016 

65,299 

Gender (Female) 

-0.37 

0.02 

<.0001 

0.0079 

65,299 

Overage for Grade 

0.88 

0.03 

<.0001 

0.0157 

65,299 

Urban residence 

1.03 

0.02 

<.0001 

0.0521 

65,299 

Suspension 






Suspensions, fallt 

1.12 

0.20 

<.0001 

0.0006 

64,348 

Suspensions, end of year 

0.66 

0.05 

<.0001 

0.0041 

65,299 

Attendance 






Attendance rate, fallt 

-3.13 

0.15 

<.0001 

0.0071 

64,348 

Attendance rate, end of year 

-6.61 

0.21 

<.0001 

0.0158 

65,299 

Mobility, Changed schools during school 
year (Yes/No) 

0.83 

0.04 

<.0001 

0.0067 

65,299 

Retained (Yes/No) 

1.38 

0.05 

<.0001 

0.0153 

65,299 

Other Variables 






Title 1 participation (Yes/No) 






School-wide 

1.20 

0.02 

<.0001 

0.0717 

65,299 

Targeted 

1.17 

0.04 

<.0001 


Exhibit Reads: students who are overage in this model are 0.88 higher in the log-odds of not being proficient in ELA MCAS than 
others. 

tlndicator was removed from final analyses, because either the direction of the coefficient of the variable was changed, or it 
was not statistically significant at an alpha level of .05 and the estimated coefficient is nearly zero. 
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Second Grade: Overview Risk Models and Final Model 

For exploratory purposes, the final model was further tested with and without school level 
characteristics. Two sources of data were used to comprehensively explore potential school-level 
characteristics: 1) student-level data in SIMS were aggregated to obtain various characteristics 
describing the set of first grade students in each school, and 2) NCES Common Core of Data (CCD): 

School Year 2007-08 were also used. In the final model, three school-level variables were included in: 1) 
the average attendance rate of students in grade 1; 2) the proportion of students who were eligible for 
free or reduced lunch at all grades; and 3) the proportion of African American students at all grades in 
each school. Other SIM-based aggregate variables were not statistically significant (i.e., the rate of 
retained students in grade 1, the rate of students in grade 1 who needed special education services, the 
rate of students receiving ELL programs in grade 1, the average number of days of suspension (in and 
out of school suspension). Furthermore, several school-level variables from CCD (regular schools vs. 
other types of school), a charter school indicator, the proportion of Hispanic students at all grades, and 
the calculated pupil/teacher ratio) from the CCD data were tested and none of them were statistically 
significant. When the comparison was made in predictive probability of proficiency by risk level between 
the random intercept models with and without the school-level characteristics, the percentages and 
numbers of students who scored below the proficiency thresholds based on the assigned risk levels 
were almost identical (17 percent vs. 18 percent among students classified as 'low risk'; 40 percent in 
both models among students classified as 'moderate risk'; and 70 percent in both models among the 
students classified as 'high risk'). Thus, the rest of analyses for the early elementary uses a random 
intercept model without school-level characteristics. 

Exhibit Grade2.3 provides an overview of our final model for Grade 2 using summary statistics of the 
analysis. To produce the predicted values of the probability of not achieving proficient in 3 rd grade ELA 
MCAS, the random effects at the school-level were accounted for, and thus the predicted probabilities 
could be different depending on school membership, conditioning on the student-level characteristics. 
The estimates in column 2 denote the expected difference in the log-odds of not being proficient in third 
grade ELA MCAS, holding constant other variables in the model. For example, students from an urban 
residence are expected to score 0.42 points higher than other students in the log-odds of not being 
proficient in ELA MCAS, holding other variables constant. Additionally, based on the result of the 
statistically significant random intercept at the school level, we can infer that the likelihood of not being 
proficient in third grade ELA MCAS also depends on school membership, holding constant student-level 
characteristics. With the exception of attendance and gender variables, all other variables are 
statistically positively associated with the recoded outcome variable (not being proficient on the third 
grade ELA MCAS). 
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Exhibit Grade2.3. Final Model - Behavioral Variables, Demographic Variables, and Other Indicators 
Random Intercept Model, Summary Statistics 


Variable 

Estimate 

S.E. 

Pr > |t| 

Behavioral variables 

Attendance rate, end of year 

-1.77 

0.24 

<.0001 

Suspensions, end of year 

0.11 

0.04 

0.0108 

Mobility - Changed schools during school year 

0.29 

0.05 

<.0001 

Retention 

0.43 

0.06 

<.0001 

Demographic variables 

Low income household- Free lunch 

0.75 

0.03 

<.0001 

Low income household- Reduced price lunch 

0.45 

0.04 

<.0001 

Special Education 

Low level of need (less than 2 hours) 

0.74 

0.04 

<.0001 

Low level of need (2 or more hours) 

1.38 

0.06 

<.0001 

Moderate level of need 

1.86 

0.04 

<.0001 

High level of need 

2.79 

0.09 

<.0001 

ELL Status 

0.80 

0.04 

<.0001 

Sex: Female 

-0.24 

0.02 

<.0001 

Urban residence 

0.42 

0.06 

<.0001 

Overage for grade 

0.20 

0.04 

<.0001 

Other variables 

School wide Title 1 

0.39 

0.06 

<.0001 

Target Title 1 

1.24 

0.04 

<.0001 


Variance of school-level intercept (standard error): 0.2307(0.0155) 
Number of observations=65,299 


Second Grade: Illustration of Levels of Risk and MCAS Outcomes Using Final Model 

The following box plot shows the distribution of test scores by increased risk, using the Final Model for 
second grade. As the risk level increases (x axis), the proficiency level appears to decrease (y axis). The 
levels of risk are defined as follows: 

• Low Risk (approximately 75% or more of students meet the outcome variable) : Intervals 1-3; 

• Moderate Risk (approximately half or more than half of the students meet the outcome 
variable) : Intervals 4-5; and 

• High Risk (approximately a third or less of the students meet the outcome variable): Intervals 6- 
9. 15 


In order to improve the prediction accuracy, a more stringent risk level threshold could have been employed; 
however, overidentification was preferred over underidentification in order to reduce the risk of excluding 
students in need of support or intervention. 
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Exhibit Grade2.4. Final Model A - Box Plot Distribution, Grade 2 

Second Grade Risk Based on Early Elementary Outcome: Proficient or Advanced on Grade 3 ELA MCAS 



123456789 

Increased risk level 


The statistics for the final model for three levels of risk (low risk, moderate risk, and high risk) are shown 
in Exhibits Grade2.5 and Grade2.6. In summary, approximately 83 percent of students who fall into the 
low risk category have met the outcome variable of proficient or higher on the third grade ELA MCAS 
(Exhibit Grade2.7). Of the students who are categorized in the moderate risk category, approximately 
59 percent of the students have met the outcome variable. Among the high risk students approximately 
30 percent met the outcome variable and 68 percent of the students scored below proficient on the 
third Grade ELA MCAS. 
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Exhibit Grade2.5. Final Model - Risk Level Based on Box Plot Distribution, Grade 2 

Total numbers of students in sample by risk levels 


Increased risk 

level 

Estimate For Frequency 

Probability of 

Risk 

Percent 

Low risk Moderate risk 

High risk 

1 

< or = 0.1 

1829 

2.80 

1829 

0 

0 

2 

< or = 0.2 

19182 

29.38 

19182 

0 

0 

3 

< or = 0.3 

13687 

20.96 

13687 

0 

0 

4 

< or = 0.4 

7499 

11.48 

0 

7499 

0 

5 

< or = 0.5 

5989 

9.17 

0 

5989 

0 

6 

< or = 0.6 

5504 

8.43 

0 

0 

5504 

7 

< or = 0.7 

4547 

6.96 

0 

0 

4547 

8 

< or = 0.8 

3297 

5.05 

0 

0 

3297 

9 

>0.8 

3765 

5.77 

0 

0 

3765 

Total 


65299 

100.0 

34698 

13488 

17113 


Exhibit Grade2.6. Final Model- Predictive Probability of Proficiency Based on Risk Level, Grade 2 


Predictive Probability of Proficiency Based on Risk Level 


Proficiency 



Below 

Proficient 


Risk Level 

Threshold 

or Above 

Total 

Low 

6,051 

28,647 

34,698 

17% 

83% 


Moderate 

5,466 

41% 

8,022 

59% 

13,488 

High 

11,370 

70% 

5,143 

30% 

17,113 

Total 

23,487 

36% 

41,812 

64% 

65,299 

100% 
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Third Grade: Analysis Results and Predicted Risk Levels 

For third grade, five models were tested to: 1) identify individual indicators of risk and 2) identify the risk 
model that is predictive of whether a rising third grade student is at risk of not meeting the outcome 
variable of achieving a score that is proficient or higher on the third grade ELA MCAS (Exhibit Grade3.1). 


Exh bit Grade3.1 Overview of Third Grade Risk Indicators 


Grade: 

3 (Using data from grade 2 students) 

Age Grouping: 

Early elementary (1st -3rd grade) 

Risk Indicators Tested: 

Behavioral variables 


• 

Suspensions, end of year 


• 

Suspensions, Fall semester 


• 

Attendance rate, end of year 


• 

Attendance rate, Fall semester 


• 

Mobility (more than one school within the school year) 


• 

Retention 


Demographic variables 


• 

Low income household- Free lunch 


• 

Low income household- Reduced price lunch 


• 

Special education level variables (4 total) 


• 

ELL status 


• 

Immigration status 


• 

Gender 


• 

Urban residence 


• 

Overage for grade 


Other individual student variables 


• 

School wide Title 1 


• 

Targeted Title 1 

Academic Goal/ 

Proficient or higher on the third grade English language arts MCAS (proxy 

Outcome Variable 16 : 

for reading by third grade) 


NOTE: A total of 68,274 observations 
characterized as proficient or above. 


included this outcome variable for the Final model. Approximately 63 percent were 
and the remaining 37 percent were less than proficient. 


Third Grade: Simple Logistics - Analysis of Individual Indicators 

In order to build the most efficient and accurate model for the EWIS, we first examined a number of 
behavioral, demographic, and other indicators tied to individual students that may be considered in the 
resulting risk model. This analysis relied on simple logistic regressions for each individual indicator. The 
single indicator analyses allowed us to evaluate the statistical significance and coefficient for each 
indicator (Exhibit Grade3.2). This analysis was used to inform the construction of the risk models tested. 


For running the statistical regression models, the outcome variable was recoded to predict the risk/likelihood of not being 
proficient or higher on the third grade ELA MCAS. 
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Exhibit Grade3.2. Simple Logistic Regression Overview, Grade 3 


Simple Logistic regression: Indiv 

dual indicators (predictor) 

Variable 

Estimate 

S.E. 

Pr > ChiSq 

R-Square 

N 

Demographic variables (Yes/No) 






Low income household- Free lunch 

1.43 

0.02 

<.0001 

0.0926 

68,274 

Low income household- Reduced price lunch 

0.94 

0.03 

<.0001 

Special education 






Low level of need (less than 2 hours) 

0.69 

0.04 

<.0001 

0.0997 

68,274 

Low level of need (2 or more hours) 

1.57 

0.04 

<.0001 

Moderate level of need 

1.89 

0.03 

<.0001 

High level of need 

2.87 

0.07 

<.0001 

ELL status 

1.49 

0.03 

<.0001 

0.0372 

68,274 

Immigration statust 

0.75 

0.05 

<.0001 

0.0029 

68,274 

Gender (Female) 

-0.37 

0.02 

<.0001 

0.0078 

68,274 

Overage for Grade 

0.99 

0.03 

<.0001 

0.0219 

68,274 

Urban residence 

1.06 

0.02 

<.0001 

0.0564 

68,274 

Suspension 






Suspensions, fall+ 

0.88 

0.17 

<.0001 

0.0005 

67,386 

Suspensions, end of year 

0.55 

0.04 

<.0001 

0.0039 

68,274 

Attendance 






Attendance rate, fallt 

-2.58 

0.15 

<.0001 

0.0044 

67,386 

Attendance rate, end of year 

-6.58 

0.21 

<.0001 

0.0154 

68,274 

Mobility, Changed schools during school year 
(Yes/No) 

0.83 

0.04 

<.0001 

0.0068 

68,274 

Retained (Yes/No) 

1.47 

0.07 

<.0001 

0.0077 

68,274 

Other Variables 






Title 1 participation (Yes/No) 






School-wide 

1.25 

0.02 

<.0001 

0.0802 

68,274 

Targeted 

1.33 

0.03 

<.0001 


Exhibit Reads: students who are overage when they were in second grade are 0.99 higher in the log-odds of not being proficient 
in ELA MCAS than others. 

tlndicator was removed from final analyses, because either the direction of the coefficient of the variable was changed, or it 
was not statistically significant at an alpha level of .05 and the estimated coefficient is nearly zero. 

Third Grade: Overview of Final Model 

Exhibit Grade3.3 provides an overview of our final model for third grade using summary statistics of the 
analysis. The estimates in column 2 denote the expected difference in the log-odds of not being 
proficient in third grade ELA MCAS, holding constant other variables in the model. For example, 
students who were retained in grade 2 are expected to score 0.25 points higher than other students in 
the log-odds of not being proficient in ELA MCAS, holding other variables constant. With the exception 
of attendance and gender variables, all other variables are statistically positively associated with the 
recoded outcome variable (not being proficient on the third grade ELA MCAS). 
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Exhibit Grade3.3. Final Model - Behavioral Variables, Demographic Variables, and Other Indicators 
Random Intercept Model, Grade 3 


Variable 

Estimate 

S.E. 

Pr > |t| 

Behavioral variables 

Attendance rate, end of year 

-1.26 

0.24 

<.0001 

Suspensions, end of year 

0.03 

0.04 

0.4172 

Mobility - Changed schools during school year 

0.40 

0.05 

<.0001 

Retention 

0.25 

0.08 

0.0023 

Demographic variables 

Low income household- Free lunch 

0.76 

0.03 

<.0001 

Low income household- Reduced price lunch 

0.47 

0.04 

<.0001 

Special Education 

Low level of need (less than 2 hours) 

0.83 

0.04 

<.0001 

Low level of need (2 or more hours) 

1.57 

0.05 

<.0001 

Moderate level of need 

2.04 

0.04 

<.0001 

High level of need 

2.83 

0.08 

<.0001 

ELL Status 

1.02 

0.04 

<.0001 

Sex: Female 

-0.22 

0.02 

<.0001 

Urban residence 

0.45 

0.06 

<.0001 

Overage for grade 

0.30 

0.03 

<.0001 

Other variables 

School wide Title 1 

0.38 

0.06 

<.0001 

Target Title 1 

1.42 

0.04 

<.0001 


Variance of school-level intercept (standard error): 0.2620(0.0166) 
Number of observations=68274 


Third Grade: Illustration of Levels of Risk and MCAS Outcomes Using Final Model 

The following box plot shows the distribution of test scores by increased risk, using the Final model. As 
the risk level increases (x axis), the proficiency level appears to decrease (y axis). The levels of risk are 
defined as follows: 

• Low Risk (approximately 75% or more of students meet the outcome variable) : Intervals 1-3; 

• Moderate Risk (approximately half or more than half of the students meet the outcome 
variable) : Intervals 4-5; and 

• High Risk (approximately a third or less of the students meet the outcome variable): Intervals 6- 
9. 
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Exhibit Grade3.4. Final Model A - Box Plot Distribution, Grade 3 

Third Grade Risk Based on Early Elementary Outcome: Proficient or Advanced on Grade 3 ELA MCAS 


< 
i 

LU 

CO 

< 

o 



3 4 5 6 7 

Increased risk level 


The statistics for the Final model's three levels of risk (low risk; moderate risk and high risk_ are shown 
in Exhibits Grade3.5 and Grade3.6. In summary, approximately 84 percent of students who fell into the 
low risk category have met the outcome variable of proficient or higher on the third grade ELA MCAS 
(Exhibit Grade3.6). Of the students who were categorized in the moderate risk category, approximately 
59 percent of the students have met the outcome variable. Among the high risk students approximately 
28 percent met the outcome variable and 72 percent of the students scored below proficient on the 
third grade ELA MCAS. 
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Exhibit Grade3.5. Final Model - Risk Level Based on Box Plot Distribution, Grade 3 


Total numbers of students in sample by risk levels 


Increased risk 

level 

Estimate For Frequency 

Probability of 

Risk 

Percent 

Low risk Moderate risk 

High risk 

1 

< or = 0.1 

3,680 

5.39 

3,680 

0 

0 

2 

< or = 0.2 

20,167 

29.54 

20,167 

0 

0 

3 

< or = 0.3 

12,503 

18.31 

12,503 

0 

0 

4 

< or = 0.4 

6,624 

9.70 

0 

6,624 

0 

5 

< or = 0.5 

5,919 

8.67 

0 

5,919 

0 

6 

< or = 0.6 

5,365 

7.86 

0 

0 

5,365 

7 

< or = 0.7 

4,673 

6.84 

0 

0 

4,673 

8 

< or = 0.8 

3,875 

5.68 

0 

0 

3,875 

9 

>0.8 

5,468 

8.01 

0 

0 

5,468 

Total 


68,274 

100.0 

36,350 

12,543 

19,381 


Exhibit Grade3.6. Final Model- Predictive Probability of Proficiency Based on Risk Level, Grade 3 


Predictive Probability of Proficiency Based on Risk Level 


Proficiency 



Below 

Proficient 


Risk Level 

Threshold 

or Above 

Total 

Low 

5,938 

30,412 

36,350 

16% 

84% 


Moderate 

5,123 

41% 

7,420 

59% 

12,543 

High 

14,001 

72% 

5,380 

28% 

19,381 

Total 

25,062 

37% 

43,212 

63% 

68,274 

100% 
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Early Elementary Risk Model Validation: Comparison of 2008-09 to 2009-10 Cohort 

In order to show the strength of the Final model in other cohorts, the following exhibits show the extent 
to which the developed risk model using the 2009-10 cohort data correctly identifies at-risk students in 
the 2008-09 cohort. 

As shown in Exhibit Early Elementary Validation. 1, while there are general similarities in the predictive 
probability of the model between the original and the validation cohorts, there is some difference in 
predictive probability at different grade levels. In looking closely at Grade 2 and Grade 3, the validation 
cohort (2008-09 school year) has a slightly higher percentage of children deemed 'low risk' who are not 
reaching proficient levels than the original cohort (23% vs. 17%). Also, the percentage of children 
identified as 'moderate risk' who fall below the threshold in the validation cohort is slightly higher than 
the percentage in the original cohort (48% vs. 41%). This may signal that the validity of the EWIS model 
for prediction could depend on the degree of the cohort variation from year to year. EWIS validations 
will be done annually, and the model adjusted if needed. 

Exhibit Early Elementary Validation. 2 shows the finalized risk model and the validation output from the 
model for grades 1 through 3, using the 2009-10 cohort and our validation cohort (2008-09). As the 
exhibit shows, the coefficients are fairly similar in magnitude and significance, and the directions of the 
coefficients are the same between the models. The consistency of the coefficients between cohorts 
implies that the selected indicators are behaving similarly in reference to our outcome variable in 
different groups. 
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Exhibit Early Elementary Validation. 1 Predictive Probability of Proficiency Original Cohort vs. 
Validation Cohort, Grades 1-3 


Predictive Probability of Proficiency Based on Risk Level 


FIRST GRADE 




Below Threshold 

Proficient or Above 


2009-10 

2008-09 

2009-10 

2008-09 

Risk Level 

cohort 

cohort 

cohort 

cohort 

Low 

6,039 

6,051 

25,921 

19,942 

18 . 90 % 

23 . 28 % 

81 . 10 % 

76 . 72 % 

Moderate 

5,714 

6,878 

8,383 

10,389 

40 . 53 % 

39 . 83 % 

59 . 47 % 

60 . 17 % 

High 

9,060 

10,828 

4,351 

4,889 

67 . 56 % 

68 . 89 % 

32 . 44 % 

31 . 11 % 

Total 

20,813 

23,757 

38,655 

35,220 

35 . 00 % 

40 . 28 % 

65 . 00 % 

59 . 72 % 

Predictive Probability o 

: Proficiency Based on Risk Level 


SECOND GRADE 



Below Threshold 

Proficient or Above 


2009-10 

2008-09 

2009-10 

2008-09 

Risk Level 

cohort 

cohort 

cohort 

cohort 

Low 

6,051 

8,045 

28,647 

26,878 

17 . 44 % 

23 . 04 % 

82.56 

76 . 96 % 

Moderate 

5,466 

40.52 

6,422 

48 . 02 % 

8,022 

59.48 

6,952 

51 . 98 % 

High 

11,970 

12,371 

5,143 

4,404 

69 . 95 % 

73 . 75 % 

30.05 

26 . 25 % 


23,487 

26,838 

41,812 

38,234 

Total 

35 . 97 % 

41 . 24 % 

64.03 

58 . 76 % 

Predictive Probability o 

: Proficiency Based on Risk Level 


THIRD GRADE 




Below Threshold 

Proficient or Above 


2009-10 

2008-09 

2009-10 

2008-09 

Risk Level 

cohort 

cohort 

cohort 

cohort 

Low 

5,938 

16 . 34 % 

8,141 

21 . 96 % 

30,412 

83 . 66 % 

28,937 

78 . 04 % 

Moderate 

5,123 

40 . 84 % 

6,251 

50 . 30 % 

7,420 

59 . 16 % 

6,177 

49 . 70 % 

High 

14,001 

14,097 

5,380 

4,396 

72 . 24 % 

76 . 23 % 

27 . 76 % 

23 . 77 % 


25,062 

28,489 

43,212 

39,510 

Total 

36 . 71 % 

41 . 90 % 

63 . 29 % 

58 . 10 % 
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Exhibit Early Elementary Validation. 2. Overview of Findings by Cohort Using Final Model, Gradel-3 



Grade 1 

Grade 2 

Grade 3 

Variable 

Original 

Cohort 

(2009-10) 

Validation 

Cohort 

(2008-09) 

Original 

Cohort 

(2009-10) 

Validation 

Cohort 

(2008-09) 

Original 

Cohort 

(2009-10) 

Validati 

on 

Cohort 

(2008- 

09) 

Behavioral variables 







Attendance rate, end of year 

-1.13*** 

-1.37*** 

-1 77 *** 

-1.06*** 

-1.26*** 

-2 29*** 

Suspensions, end of year 

0.18** 

0 . 22 ** 

0 . 11 ** 

0.14** 

0.03 

0.14** 

Mobility - Changed schools during school year 

0.33*** 

0.26*** 

q 29 *** 

0 42*** 

0.40*** 

0.50*** 

Retention 

- 

- 

0.43*** 

0.37*** 

0.25** 

0.34*** 

Demographic variables 







Low income household- Free lunch 

0.80*** 

0 go*** 

0.75*** 

0 . 88 *** 

0.76*** 

0.84*** 

Low income household- Reduced price lunch 

0 47 *** 

0.61*** 

0.45*** 

q 57 *** 

0 47 *** 

0.60*** 

Special Education 







Low level of need (less than 2 hours) 

0 . 66 *** 

q 72 *** 

q 74 *** 

0.76*** 

0.83*** 

0 gg* * * 

Low level of need (2 or more hours) 

1.25*** 

1.32*** 

1.38*** 

^ 52*** 

57 *** 

1.61*** 

Moderate level of need 

1.63*** 

1.65*** 

1 . 86 *** 

1 3i*** 

2 04 *** 

^ oo*** 

High level of need 

2.61*** 

2.35*** 

2 79 *** 

2.53*** 

2.83*** 

2 56 *** 

ELL Status 

0.62*** 

0.56*** 

0.80*** 

0 . 66 *** 

1 02 *** 

0.96*** 

Sex: Female 

-0.28*** 

-0 17 *** 

-0.24*** 

-0.13*** 

-0 22 *** 

_o ^ 1 *** 

Urban residence 

0.33*** 

0.37*** 

q 42*** 

0 44 *** 

0.45*** 

0.50*** 

Overage for grade 

q 22 *** 

- 0 . 12 * 

0 . 20 *** 

0 24 *** 

0.30*** 

0 29 *** 

Other variables 







School wide Title 1 

0 3o*** 

q 29 *** 

0.39*** 

0.35*** 

0.38*** 

0.39*** 

Target Title 1 

1 02 *** 

q gg** * 

1 24 *** 

1 24 *** 

1 42 *** 

\ 54 *** 


Significant at 10%, **Significant at 5%, ***Significant at 1% 
- variable not included in model 
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Late Elementary Age Group (Fourth through Sixth Grade) 

The Late Elementary Age Group encompasses fourth through sixth grade, using data from third, fourth 
and fifth graders. Within the age group indicators of risk were tested at each grade level based on the 
outcome variable of scoring proficient or higher on a combined proficiency of 6th Grade English 
Language Arts MCAS and 6th Grade Math MCAS. 

Potential Indicators 

In the Late Elementary Age Group, the indicators tested included behavioral, demographic, other 
variables, and the MEPA levels and proficiency in both ELA and math MCAS. Behavioral indicators are 
mutable and considered manifestations of student behavior (e.g., attendance, suspensions). 
Demographic indicators are tied to who the child is, and are not necessarily based on a student's 
behavior (although some of these, such as low income household, may change over time). Other 
individual student indicators are focused on characteristics related to the community in which the 
student resides and the type of services the student receives. The late elementary analysis brings in skill 
assessments using MEPA levels and math and English MCAS proficiency levels, which results in 
substantial improvement of prediction accuracy. Exhibit Late Elementary. 1 provides a summary of the 
indicators that were tested in the late elementary grades. 17 


Exhibit Late Elementary.l. Indicator Definitions, by Type 


Type 

Indicator 

Definition 

Corresponding 

Data Source 

Outcome Variable 



Sixth Grade English 
Language Arts and 
Math MCAS 

Binary variable: 1= Proficient or above proficient in both 
Math and English; 0=Warning or needs improvement in 
either Math or English 

Indicates students who achieve a proficient (or higher) 
or below proficient score on the Sixth Grade ELA and 

Math MCAS 

MCAS 2010 data 

variable name: EPERF2 

MPERF2 

Behavioral Variable 



Attendance 

Continuous variable: Attendance rate, end of year- 
number of days in attendance over the number of days 
in membership 

SIMS DOE045 

SIMS DOE046 


Suspension 

Continuous variable: Suspensions, end of year - number 
of days in school suspension plus number of days out of 
school suspension 

SIMS DOE017 

SIMS DOE018 


Retention 18 

Binary variable: Based on whether child is listed as grade 

1 status in both 2007 and 2008 Fall data l=Retained; 

0=Not retained 

SIMS DOE016 


Mobility 

Binary variable: l=School code changes from beginning 
of school year to end of school year; 0= School code is 
the same at beginning and end of school year 

SIMS 8 digit school 
identifier 

Demographic variable 



Gender 

Binary variable: l=Female; 0=Male 

SIMS DOE009 


Low income 

household - Free 

Binary variable: l=Free lunch eligible; 0= not eligible 

SIMS DOE019 


17 The table includes all variables tested in the Late Elementary Age Group, but there may be variation in which of 
these were tested in individual grades. 

18 Retention is defined from fall to fall. 
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Type 

Indicator 

Definition 

Corresponding 

Data Source 


lunch 




Low income 

household - 
Reduced price lunch 

Binary variable: l=Reduced lunch recipient; 0= Not 
eligible for reduced price lunch 

SIMS DOE019 


Former LEP 

Binary variable: 1= Former LEP student; 0=Current or no 
LEP status 

e.g., MCAS 2008 data 
variable for 2008-09 

cohort data name: 
LEPFLEP_OFF minus 
LEP_OFF 


ELL program 

Binary variable: 1= sheltered English Immersion (SEI) or 
2-way bilingual or other; 

0 = opt out, no program 

SIMS DOE014 


Overage for grade 

Binary variable: l=Age of child is equal to or greater than 
one year expected grade level age as of September 1 in a 
given calendar year; 0= Age of child is less than one year 
more than expected grade level age (e.g. a student 10 or 
older on September 1 st of their 4 th grade year is overage) 

SIMS DOE006 


Immigration Status 

Binary variable: 1= Student is an immigrant under the 
federal definition; 0=Student is not an immigrant 

SIMS DOE022 


Urban residence 

Binary variable: l=Student lives in an urban area 19 ; 0= 
Student does not live in one of the specified urban areas 

SIMS DOE014 


Special Education - 
Level of Need 

Special Education - Multiple indicators 

• Dummy variable: Low level of need (less than 2 
hours) is equal to 1; otherwise 0. 

• Dummy variable: Low level of need (2 or more 
hours) is equal to 1; otherwise 0. 

• Dummy variable: Moderate level of need is 
equal to 1; otherwise 0. 

• Dummy variable: High level of need is equal to 

1; otherwise 0. 

SIMS DOE038 

Other Individual Student Variable 



Title 1 participation 

Binary variables: 

• Targeted Title 1, Binary variable: 1= Any type of 
targeted Title 1 participation; 0= Not included in 
targeted Title I 20 

• School -wide Title 1, Binary variable: 1= School- 
wide Title 1; 0= Not school-wide Title 1 

SIMS DQE020 


19 Specified urban areas: Boston, Brockton, Cambridge, Chelsea, Chicopee, Everett, Fall River, Fitchburg, 
Framingham, Haverhill, Holyoke, Lawrence, Leominster, Lowell, Lynn, Malden, New Bedford, Pittsfield, Quincy, 
Revere, Somerville, Springfield, Taunton, Worcester. This reflects urban areas for years tested. 

20 There is only one possible outcome per student for the Title I variable, so if they are elected as school-wide Title 
I they cannot be considered targeted and vice versa according to the data. 
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MEPA Levels 



Massachusetts English 
Proficiency 

Assessment (MEPA) 

Binary variable: 

• Beginner level to Intermediate level is equal to 1; 
otherwise 0. 21 

MEAP Spring 2008 data 
variable name: 

Pi 

MCAS Proficiency Levels 



MCAS Proficiency 
levels in Math and 
English 

Multiple indicators 

• Math 22 

o Dummy variable: Warning is equal to 

1; otherwise 0. 

o Dummy variable: Needs improvement 

is equal to 1; otherwise 0. 

o Dummy variable: Proficient is equal to 

1; otherwise 0. 

• English 23 

o Dummy variable: Warning is equal to 

1; otherwise 0. 

o Dummy variable: Needs improvement 

is equal to 1; otherwise 0. 

o Dummy variable: Proficient is equal to 

1; otherwise 0. 

e.g., MCAS 2008 data 
variable for 2008-09 

cohort data 

name: 

EPERF2 MPERF2 


Analysis Methods and Strategies 

To identify the model that most accurately predicts risk of not achieving proficiency on sixth grade ELA 
and Mathematics MCAS, we conducted multiple analyses. For prediction of the sixth grade ELA and 
math MCAS proficiency, a separate analysis was conducted in each grade taking into account the 
students information from the previous year to provide a risk level for students as they enter the next 
year : fourth grade (using students' grade 3 data ), fifth grade (using students' grade 4 data) and sixth 
grade (using students' grade 5 data). 

For data analysis, we focused on 2009-10 sixth grade cohort that include students with valid sixth grade 
ELA and math MCAS performance scores, and SIMS data in 2006-07 through 2009-10 were analyzed to 
identify the predictive indicators in each grade (see Exhibit Late Elementary. 2). 


21 Originally multiple indicators of MEPA levels (Beginner, Early intermediate, Intermediate, Transiting to regular classes) were 
tested. However, due to small sample in individual MEPA levels with district data, final model aggregates MEPA levels beginner 
to intermediate as a single indicator, leaving transiting to regular classes and non-MEPA as 0 for this variable. The benefit of this 
strategy is that this indicator fits in the EWIS models with the current MEPA levels having 5 categories (Transiting to regular 
classes I and Transiting to regular classes II). Thus, the binary indicator of MEPA levels was used for the rest of EWIS models in 
middle and high school age groups. - 

22 Above proficient left out as reference category 

23 Above proficient left out as reference category 
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Exhibit Late Elementary.2. Numbers of students and schools by data source 


6 th grade Proficiency in both ELA & Math MCAS 


Source Data 

Warning or needs 

Proficient or 

# Students 


improvement in 

Above in ELA 



either Math or 

and Math 



English 



Grade 6 in 2009-10 

33,186 (46%) 

38,757 (54%) 

71,943 

Grade 3 in 2006-07 

(used to create 4 th grade model) 

29,249 (45%) 

36,248 (55%) 

65,497 

Grade 4 in 2007-08 

(used to create 5 th grade model) 

28,830 (44%) 

35,960 (56%) 

64,790 

Grade 5 in 2008-09 

(used to create 6 th grade model) 

29,847 (45%) 

36,747 (55%) 

66,594 


The following strategies were employed in grade level analyses. 

• First, in order to build an efficient and accurate model for the EWIS, we first examined a number 
of behavioral, demographic, and other individual student variables that may be considered in 
the resulting risk model. This analysis relied on simple logistic regressions for each individual 
indicator. The individual indicator analyses allowed us to evaluate the statistical significance 
and coefficient for each indicator. This analysis was used to inform the construction of the risk 
models tested. 

• Then, based on the results of the simple logistic regression models, a series of analysis were 
conducted - 

o Student behavioral variables only; 

o Demographic variables along with the behavioral variables from the previous model; 
o Demographic variables, behavioral variables, and individual student variables including 
the availability of school wide and targeted Title I; 
o Demographic variables, behavioral variables, individual student variables including the 
availability of school wide and targeted Title I, and MEPA levels; 
o Demographic variables, behavioral variables, and individual student variables including 
the availability of school wide and targeted Title I, MEPA levels, and MCAS proficiency 
levels. 

Additionally, multi-level logistic regression, to account for the clustering of students within schools, and 
allowed the level-1 intercept to be random, was tested for grade 5 and 6 models. The EWIS models were 
able to be updated to use more recent MCAS scores than MCAS data used in model testing by AIR. Due 
to time constraints and issues with transferring the statistical syntax, ESE made the decision to update 
the most predictive logistical regressions with the more recent MCAS scores than use multi-level 
logistical regression with older data. Future iterations of the late elementary models may utilize multi- 
level logistical regression. 

Additionally, the model created using information for the third grade year was initially part of the early 
elementary model with the outcome for being proficient or advanced for on the grade 3 ELA MCAS. In 
order for this predictive model to be of value, it was built using fall indicators and would provide a mid- 
year indicator for students so it could be provided to educators before those students took their grade 3 
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MCAS (the outcome event). Upon further reflection, ESE decided to use the full year of third grade 
student information to create EWIS model and designate risk levels for rising fourth grade students tied 
to the Late Elementary Model Outcome, Proficient or Advanced on the grade 6 ELA and Math MCAS. 
Therefore, the model for fifth grade and sixth grade were finalized before the model for fourth grade 
was finalized. Testing for possible interaction terms were done for the later grades and are discussed in 
those sections. 
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Fourth Grade: Analysis Results and Predicted Risk Levels 

In fourth grade, several models were tested to: 1) identify individual indicators of risk and 2) identify the 
risk model that is predictive of whether a rising fourth grade student is at risk of not meeting the 
academic goal of achieving a score that is proficient or higher on both the sixth grade English language 
arts (ELA) and Math Massachusetts Comprehensive Assessment System (MCAS) (Exhibit Grade4.1). 


Exhibit Grade4.1 Overview of Fourth Grade Risk Indicators 


Grade: 

4 (using data form Grade 3) 

Age Grouping: 

Late elementary (4 th thru 6 th grade) 

Risk Indicators Tested: 

Behavioral variables 

• Suspensions, fall 

• Suspensions, end of year 

• Attendance rate, fall 

• Attendance rate, end of year 

• Mobility (more than one school within the school year) 

• Retention 

Demographic variables 

• Low income household- Free lunch 

• Low income household- Reduced price lunch 

• Special education level variables (4 total) 

• ELL status 

• Former LEP 

• Immigration status 

• Gender 

• Urban residence 

• Overage for grade (9 or older as of Sept 1 of 3 rd grade year) 

Other individual student variables 

• School wide Title 1 

• Targeted Title 1 

MEPA levels 24 

• Low Levels 

3 rd Grade MCAS Proficiency Levels 

• Math 

o Warning 
o Needs Improvement 
o Proficient 

• English 

o Warning 
o Needs Improvement 
o Proficient 

Academic Goal/ 
Outcome Variable: 25 

Proficient or higher on the sixth grade ELA and Math MCAS; Not 
proficient on either is equal to zero for this outcome 


NOTE: A total of 62,774 observations included this outcome variable for the final model. Approximately 56 percent were 
characterized as proficient or advanced on both the ELA and Math MCAS, and the remaining 44 percent were less than 
proficient. 


24 Due to small sample in individual MEPA levels, final model aggregates MEPA levels beginner to intermediate as a single 
indicator, leaving transitioning to regular classes and non-MEPA as 0 for this variable. 

25 For running the statistical regression models, the outcome variable was recoded to predict the risk/likelihood of not being 
proficient or higher on the sixth third grade ELA and/or Math MCAS. 
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Fourth Grade: Simple Logistics - Analysis of Individual Indicators 

In order to build the most efficient and accurate model for the EWIS, we first examined a number of 
behavioral, demographic, and other indicators from the Fall semester tied to individual students that 
may be considered in the resulting risk model. This analysis relied on simple logistic regressions for each 
individual indicator. The single indicator analyses allowed us to evaluate the statistical significance and 
coefficient for each indicator (Exhibit Grade4.2). This analysis was used to inform the construction of 
the risk models tested. 

Exhibit Grade4.2. Simple Logistic Regression Overview, Grade 4 


Simple Logistic regression: Individual indicators (predictor) 


Variable 

Estimate 

S.E. 

Pr > ChiSq 

R-Square 

N 

Demographic variables (Yes/No) 






Low income household- Free lunch 

1.57 

0.02 

<.0001 

0.103 

65,497 

Low income household- Reduced price lunch 

0.99 

0.03 

<.0001 

Special education 






Low level of need (less than 2 hours) 

.965 

0.05 

<.0001 



Low level of need (2 or more hours) 

1.81 

0.05 

<.0001 

0.121 

65,497 

Moderate level of need 

2.13 

0.04 

<.0001 

High level of need 

3.37 

0.10 

<.0001 



Immigration statust 

0.37 

0.05 

<.0001 

0.001 

65,497 

Sex: Female 

-0.28 

0.02 

<.0001 

0.003 

65,497 

ELL status t 

1.15 

0.04 

<.0001 

0.017 

65,497 

Former LEP studentt 

0.30 

0.05 

<.0001 

0.001 

65,497 

Overage for grade 

1.30 

0.03 

<.0001 

0.040 

65,497 

Urban residence 

1.06 

0.02 

<.0001 

0.056 

65,497 

Suspension 






Suspensions, fallt 

1.14 

0.21 

<.0001 

0.001 

65,497 

Suspensions, end of year 

1.10 

0.06 

<.0001 

0.010 

65,497 

Attendance 






Attendance rate, fallt 

-3.68 

0.17 

<.0001 

0.009 

65,497 

Attendance rate, end of year 

-9.65 

0.25 

<.0001 

0. 026 

65,497 

Mobility - Changed schools during school year (Y/N) 

1.05 

0.04 

<.0001 

0.010 

65,497 

Retained (Yes/No) + 

1.95 

0.11 

<.0001 

0. 008 

65,497 

Title 1 participation (Yes/No) 






School-wide 

1.28 

0.02 

<.0001 

0. 086 

65,497 

Targeted 

1.54 

0.04 

<.0001 

MEPA Levels (Yes/No) 






Low level 

2.17 

0.06 

<.0001 

0.027 

65,497 

3 rd Grade MCAS Proficiency Levels 






MATH 






Warning 

5.72 

0.06 

<.0001 



Needs Improvement 

3.57 

0.04 

<.0001 

0.351 

65,497 

Proficient 

1.79 

0.04 

<.0001 



ENGLISH 






Warning 

5.84 

0.08 

<.0001 



Needs Improvement 

3.74 

0.05 

<.0001 

0.306 

65,497 

Proficient 

1.80 

0.05 

<.0001 




tlndicator was removed from final analyses, because either the direction of the coefficient of the variable was changed, or it 
was not statistically significant at an alpha level of .05 and the estimated coefficient is nearly zero. 
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Fourth Grade Overview of Risk Models and Final Model 

Several risk models were tested and additional predictive probability of the model was balanced with 
model complexity. Several external factors as described in the previous section influenced the final 
model for this grade level. Exhibit Grade4.3 provides an overview of five of these models. Model 5 is 
the final Fourth grade EWIS model. 

Exhibit Grade 4.3. Overview of Findings by Model, Grade 4 



Logistic Regression Analysis 

Variable 

Model 1 

Model 2 

Model 3 

Model 4 

Final Model 

Model 5 

Behavioral variables 






Attendance, Fallt 

-1.15*** 





Attendance, end of year 

-8.29*** 

-5.54*** 

-5.55*** 

-5.57*** 

-3.96*** 

Suspensions, Fallt 

0.15 





Suspensions, end of year 

0.93*** 

0.39*** 

0.37*** 

0.37*** 

0.21*** 

Mobility, Changed schools by the end of year 

.933*** 

0.45*** 

0.46*** 

0.45*** 

0.16** 

Retentiont 

1.81*** 

0.55*** 

0.52*** 

0.51*** 

.48*** 

Demographic variables 






Low income household- Free lunch 


1.11*** 

0.98*** 

0.97*** 

0.52*** 

Low income household- Reduced price lunch 


0.78*** 

0.67*** 

0.67*** 

0.35*** 

Special education status 






Low level of need (less than 2 hours) 


1.09*** 

1.07*** 

1.07*** 

0.59*** 

Low level of need (2 or more hours) 


1.72*** 

1.700*** 

1.70*** 

0.95*** 

Moderate level of need 


2.09*** 

2.13*** 

2.12*** 

1.11*** 

High level of need 


3.09*** 

3.11*** 

3.12*** 

1.75*** 

ELL Statust 


0.43*** 

0.31*** 

-0.14* 


Former LEPt 


-0.22** 




Immigration statust 


-0.37** 




Sex: Female 


-0.02 

-0.03 

-0.02 

-0.14*** 

Urban residence 


0.47*** 

0.25*** 

0.25*** 

0.03 

Overage for grade 


0.57*** 

0.55*** 

0.54*** 

0.42*** 

Other Indicators 






School wide Title 1 



0.57*** 

q 49*** 

0.36*** 

Targeted Title 1 



1.50*** 

1.50*** 

0.65*** 

MEPA Levels 26 (Yes/No) 






Low Level 




1,31*** 

.12* 

3 rd Grade MCAS Proficiency Levels 






MATH 






Warning 





3.79*** 

Needs Improvement 





2.45*** 

Proficient 





1.24*** 

ENGLISH 






Warning 





2.71*** 

Needs Improvement 





2.01*** 

Proficient 





1.04*** 

r A 2 

0.043 

0. 223 

0.241 

0.245 

0.424 


Due to small sample sizes in categories, final model groups beginning to intermediate levels into one category, 
leaving transitioning to regular classes and non-MEPA as the reference group. 
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Exhibit Grade4.4 provides an overview of our Final model for Grade 4 using summary statistics of the 
analysis. The estimates in column 2 denote the expected difference in the log-odds of not being 
proficient in third grade ELA MCAS, holding constant other variables in the model. With the exception 
of attendance and gender variables, all other variables are statistically positively associated with the 
recoded outcome variable (not being proficient on the third grade ELA MCAS). For example, students 
who were overage for their grade are expected to score 0.42 points higher than other students in the 
log-odds of not being proficient on the sixth grade Math and ELA MCAS, holding other variables 
constant. 

Exhibit Grade4.4. Final Model - Behavioral Variables, Demographic Variables, Other Variables, MEPA 
Levels, and MCAS 


Variable 

Estimate 

S.E. 

Pr > 1 1 1 

Behavioral variables 

Attendance rate, end of year 

-3.96 

0.35 

<.0001 

Suspensions, end of year 

0.21 

0.06 

<.0001 

Mobility - Changed schools during school year 

0.16 

0.07 

0.025 

Demographic variables 

Low income household- Free lunch 

0.52 

0.07 

<.0001 

Low income household- Reduced price lunch 

0.35 

0.05 

<.0001 

Special Education 

Low level of need (less than 2 hours) 

0.59 

0.06 

<.0001 

Low level of need (2 or more hours) 

0.95 

0.06 

<.0001 

Moderate level of need 

1.11 

0.05 

<.0001 

High level of need 

1.75 

0.13 

<.0001 

Sex: Female 

-0.14 

0.02 

<.0001 

Urban residence 

0.03 

0.02 

0.4 

Overage for grade 

0.42 

0.04 

<.0001 

Other variables 

School wide Title 1 

0.12 

0.09 

<.0001 

Target Title 1 

0.65 

0.05 

<.0001 

MEPA Levels 

Low level (Beginner to intermediate) 

0.12 

0.01 

0.01 

3 rd Grade MCAS Proficiency Levels 

MATH 

Warning 

3.79 

0.07 

<.0001 

Needs Improvement 

2.45 

0.05 

<.0001 

Proficient 

1.24 

0.04 

<.0001 

ENGLISH 

Warning 

2.71 

0.09 

<.0001 

Needs Improvement 

2.01 

0.05 

<.0001 

Proficient 

1.04 

0.05 

<.0001 

R A 2=0.424 





Number of observations=62,774 
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Fourth Grade: Illustration of Levels of Risk and MCAS Outcomes Using Final Model 

The following box plot shows the distribution of test scores by increased risk, using the Final Model. As 
the risk level increases (x axis), the proficiency level appears to decrease (y axis). 


Exhibit Grade4.5. Final Model- Box Plot Distribution, Grade 4 Risk Level and 6 
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Exhibit Grade4.6. Final Model- Box Plot Distribution, Grade 4 Risk Level and 6 
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Based on analysis, the levels of risk were defined as follows: 

• Low Risk (approximately 75% or more of students meet the outcome variable): Intervals 1-3; 

• Moderate Risk (approximately half or more than half of the students meet the outcome 
variable): Intervals 4-5; and 

• High Risk (approximately a third or less of the students meet the outcome variable): Intervals 6- 
9. 


The statistics for the final model's three levels of risk (low risk; moderate risk and high risk) are shown in 
Exhibits Grade4.6 and Grade4.7. In summary, approximately 88 percent of students who fall into the 
low risk category have met or exceeded the proficiency level in both ELA and math on the grade 6 
MCAS. Of the students who are categorized in the moderate risk category, approximately 55 percent of 
the students have met the outcome variable. Among the high risk students approximately 16 percent 
met the outcome variable and 84 percent of the students scored below proficient on the six grade 
proficiency in ELA and/or math. 

Note that as assessment data (MCAS test scores) were included in the risk model, the prediction 
accuracy rates in the late elementary group analysis increased, and the actual numbers of students that 
were incorrectly identified (either students classified as 'low risk' who actually did not meet the 
threshold or students classified as 'high risk' who actually met or exceeded the proficiency threshold) 
decreased in comparison with the numbers from the early elementary age group analyses. 

Exhibit Grade4.6. Final Model - Risk Level Based on Box Plot Distribution, Grade 4 


Total numbers of students in sample by risk levels 


Increased risk 

level 

Estimate For 
Probability of 
Risk 

Frequency 

Percent 

Low risk 

Moderate risk 

High risk 

1 

< or = 0.1 

13,287 

21.2 

13,287 

0 

0 

2 

< or = 0.2 

10,907 

17.4 

10,907 

0 

0 

3 

< or = 0.3 

4,334 

6.9 

4,334 

0 

0 

4 

< or = 0.4 

5,172 

8.2 

0 

5,172 

0 

5 

< or = 0.5 

3,694 

5.9 

0 

3,694 

0 

6 

< or = 0.6 

2,683 

4.3 

0 

2,683 

0 

7 

< or = 0.7 

4,260 

6.8 

0 

0 

4,260 

8 

< or = 0.8 

3,364 

5.4 

0 

0 

3,364 

9 

>0.8 

15,073 

24.0 

0 

0 

15,073 

Total 


62,774 

100.0 

28,528 

11,549 

22,697 
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Exhibit Grade4.7. Final Model- Predictive Probability of Proficiency Based on Risk Level, Grade 4 


Predictive Probability of Proficiency Based on Risk Level 


Proficiency 



Below 

Proficient 


Risk Level 

Threshold 

or Above 

Total 

Low 

3,375 

11 . 83 % 

25,153 

88 . 17 % 

28,528 

Moderate 

5,270 

45 . 63 % 

6,279 

54 . 37 % 

11,549 

High 

19,121 

84 . 24 % 

3,576 

15 . 76 % 

22,697 


27,766 

35,008 

62,774 

Total 

44 . 23 % 

55 . 77 % 

100 % 
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Fifth Grade: Analysis Results and Predicted Risk Levels 

In fifth grade, several models were tested to: 1) identify individual indicators of risk and 2) identify the 
risk model that is most predictive of whether a rising fifth grade student is at risk of not meeting the 
combined outcome variable of achieving a score that is proficient or higher in both the sixth grade 
English language arts (ELA) and Mathematics (Math) MCAS (Exhibit Grade5.1). 


Exh 


bit Grade5.1 Overview of Fifth Grade Risk Indicators 

Grade: 

5 (using fourth grade information) 

Age Grouping: 

Late elementary (4 th thru 6 th grade) 

Risk Indicators Tested: 

Behavioral variables 


• 

Suspensions, fall 


• 

Suspensions, end of year 


• 

Attendance rate, fall 


• 

Attendance rate, end of year 


• 

Mobility (more than one school within the school year) 


• 

Retention 


Demographic variables 


• 

Low income household- Free lunch 


• 

Low income household- Reduced price lunch 


• 

Special education level variables (4 total) 


• 

ELL status 


• 

Former LEP 


• 

Immigration status 


• 

Gender 


• 

Urban residence 


• 

Overage for grade (age 10 or older by Sept 1 st of 4 th grade year) 


Other 

individual student variables 


• 

School wide Title 1 


• 

Targeted Title 1 


MEPA 

levels 27 


• 

Beginner 


• 

Early Intermediate 


• 

Intermediate 


• 

Transitioning to regular classes 


4 Grade MCAS Proficiency Levels 


• 

Math 



o Warning 
o Needs Improvement 
o Proficient 


• 

English 



o Warning 
o Needs Improvement 
o Proficient 

Academic Goal/ 

Proficient or higher on the sixth grade ELA and Math MCAS; Not 

Outcome Variable: 28 

proficient on either is equal to zero for this outcome 


Due to small sample in individual MEPA levels, final model aggregates MEPA levels beginner to intermediate as a single 
indicator, leaving transitioning to regular classes and non-MEPA as 0 for this variable. 

28 For running the statistical regression models, the outcome variable was recoded to predict the risk/likelihood of not being 
proficient or higher on the sixth third grade ELA and/or Math MCAS. 
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NOTE: A total of 67,403 observations included this combined outcome variable for the final model. Approximately 56 percent 
were characterized as proficient or above in both math and ELA, and 44 percent were less than proficient in one or both. 

Fifth Grade: Simple Logistics - Analysis of Individual Indicators 

In order to build an efficient and accurate model for the EWIS, we first examined a number of 
behavioral, demographic, other indicators, and MEPA and MCAS, tied to individual students that may be 
considered in the resulting risk model. This analysis relied on simple logistic regressions for each 
individual indicator. The single indicator analyses allowed us to evaluate the statistical significance and 
coefficient for each indicator (Exhibit Grade5.2). This analysis was used to inform the construction of 
the risk models tested (Exhibit Grade5.3). 

Exhibit Grade5.2. Simple Logistic Regression Overview, Grade 5 


Simple Logistic regression: Individual indicators (predictor) 


Variable 

Estimate 

S.E. 

Pr > ChiSq 

R-Square 

N 

Demographic variables (Yes/No) 






Low income household- Free lunch 

1.56 

0.02 

<.0001 

0.1017 

64,790 

Low income household- Reduced 
price lunch 

0.97 

0.03 

<.0001 

Special education 






Low level of need (less than 2 hours) 

1.16 

0.05 

<.0001 

0.1417 

64,790 

Low level of need (2 or more hours) 

1.85 

0.05 

<.0001 

Moderate level of need 

2.33 

0.04 

<.0001 

High level of need 

3.55 

0.11 

<.0001 

Immigration statust 

0.15 

0.07 

0.032 

0.0001 

64,790 

Sex: Female 

-0.21 

0.02 

<.0001 

0.0027 

64,790 

ELL status 

1.45 

0.05 

<.0001 

0.0175 

64,790 

Former LEP studentt 

0.32 

0.05 

<.0001 

0.0006 

64,790 

Overage for grade 

1.33 

0.04 

<.0001 

0.0422 

64,790 

Urban residence 

1.06 

0.02 

<.0001 

0.0550 

64,790 

Suspension 






Suspensions, fallt 

1.74 

0.23 

<.0001 

0.0016 

64,500 

Suspensions, end of year 

1.05 

0.05 

<.0001 

0.0123 

64,790 

Attendance 






Attendance rate, fallt 

-4.02 

0.18 

<.0001 

0.0086 

64,500 

Attendance rate, end of year 

-10.48 

0.25 

<.0001 

0. 0299 

64,790 

Mobility - Changed schools during 
school year (Yes/No) 

1.15 

0.06 

<.0001 

0.0073 

64,790 

Retained (Yes/No) 

2.26 

0.17 

<.0001 

0. 0043 

64,790 

Title 1 participation (Yes/No) 






School-wide 

1.27 

0.02 

<.0001 

0. 0871 

64,790 

Targeted 

1.70 

0.04 

<.0001 

MEPA Levels (Yes/No) 






Low level (Beginner to intermediate) 

2.68 

0.09 

<.0001 

0.021 

64,790 
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Simple Logistic regression: Individual indicators (predictor) 


Variable 

Estimate 

S.E. 

Pr > ChiSq 

R-Square 

N 

4 th Grade MCAS Proficiency Levels 






MATH 






Warning 

6.08 

0.08 

<.0001 

0.355 

64,790 

Needs Improvement 

3.02 

0.03 

<.0001 

Proficient 

1.03 

0.03 

<.0001 

ENGLISH 






Warning 

4.84 

0.06 

<.0001 

0.292 

64,790 

Needs Improvement 

2.25 

0.04 

<.0001 

Proficient 

.331 

0.04 

<.0001 


Exhibit Reads: students with a high level of need are 3.55 higher in the log-odds of not being proficient in ELA and/or Math 
MCAS than others. 

Fifth Grade: Risk Models Overview and Final Model 

In total, we tested several potential risk models 29 (Exhibit Grade5.2). After testing the statistical 
significance of the interaction of two predictors respectively, the interaction terms were not included in 
the Final model, and three interaction effects were statistically significant - interactions between 
mobility and retention, between gender and suspensions, and between overage and urban residence. 
However, adding three more interaction effects into the final model did not much improve the 
prediction accuracy, and the proportions of students who did not meet the outcome threshold in each 
designated risk level were almost identical between the Final model and the model with interaction 
terms (see Exhibit Grade5.7 based on the Final model). Moreover, the coefficient of the interaction 
between mobility and retention was statistically negative, and did not make sense, because this implies 
that students who were retained and moved the school during the school year were less likely to fall off 
the track. Therefore, model 5 was accepted as the final fifth grade risk model, because it is as predictive 
as the model with some interaction terms and because it is a simpler model. 


For a subset of students whose MCAS measures were missing, the decision was made that such students will be 
automatically flagged as 'moderate risk', and if possible teachers or school personnel are recommended to further 
determine whether or not the students need supplementary preventions based on local information. 
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Exhibit Grade5.3. Overview of Findings by Model, Fifth Grade 



Logistic Regression Analysis 

Variable 

Model 1 

Model 2 

Model 3 

Model 4 

Final Model 

Model 5 

Behavioral variables 






Attendance, Fall t 

-0.84*** 





Attendance, end of year 

_9 

-5 90*** 

-5.90*** 

-5.88*** 

-4.78*** 

Suspensions, Fallt 

0.14 





Suspensions, end of year 

0 39* ** 

0.38*** 

0 37* ** 

0 37* * * 

0.09** 

Mobility, Changed schools by the end of 
year 

1 02*** 

Q 92*** 

0.56*** 

0.56*** 

Q 97*** 

Retention 

2 13* ** 

0.85*** 

0.78*** 

0.80*** 

0.82*** 

Demographic variables 






Low income household- Free lunch 


2 21*** 

0 93*** 

0 93*** 

q 97*** 

Low income household- Reduced price 
lunch 


0 73*** 

0.61*** 

0.61*** 

0 3 1** * 

Special education status 






Low level of need (less than 2 hours) 


1 23*** 

2 22*** 

2 22*** 

0.63*** 

Low level of need (2 or more hours) 


1 79*** 

1 79*** 

1 73* * * 

0.95*** 

Moderate level of need 


2 28*** 

2 33*** 

2 32*** 

1.26*** 

High level of need 


3 19*** 

3 29*** 

3 28*** 

1 3i*** 

ELL Statust 


q 77*** 

0.61*** 

0.50*** 


Former LEP+ 


-0.22** 




Immigration statust 


-0.61*** 




Sex: Female 


-0.01 

-0.01 

-0.01 

-0.02 

Urban residence 


Q 4g*** 

0.28*** 

0.28*** 

q 2i*** 

Overage for grade 


0 57* ** 

Q 99* ** 

0.54*** 

0.40*** 

Other Indicators 






School wide Title 1 



q 4y*** 

q 4y*** 

0.38*** 

Targeted Title 1 



1 70*** 

2 70* * * 

0 95*** 

MEPA Levels 30 (Yes/No) 






Low level (Beginner to intermediate) 




2 42*** 

2 2i*** 

4 th Grade MCAS Proficiency Levels 






MATH 






Warning 





4 37*** 

Needs Improvement 





2.46*** 

Proficient 





0 95*** 

ENGLISH 







Due to small sample sizes in categories, final model groups beginning to intermediate levels into one category, 
leaving transitioning to regular classes and non-MEPA as the reference group. 
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Logistic Regression Analysis 

Variable 

Model 1 

Model 2 

Model 3 

Model 4 

Final Model 

Model 5 

Warning 





1.60*** 

Needs Improvement 





0.64*** 

Proficient 





0.28*** 

r A 2 

0.0460 

0. 2412 

0.2601 

0.2619 

0.437 


tlndicator was removed from final analyses, because either the direction of the coefficient of the variable was changed, or it 
was not statistically significant at an alpha level of .05 and the estimated coefficient is nearly zero. 

* Significant at 10%, **Significant at 5%, ***Significant at 1% 


Exhibit Grade5.4 provides the summary statistics for the Final model. The estimates in column 2 denote 
the expected difference in the log-odds of not being proficient in our combined outcome variable— sixth 
grade ELA and/or Math MCAS, holding constant other variables in the model. With the exception of 
attendance and gender variables, all other variables are statistically positively associated with the 
recoded outcome variable. For example, students that are overage for when they were in fourth grade 
are expected to score 0.40 points higher than other students in the log-odds of not being proficient in 
ELA and/or Math MCAS, holding other variables constant. 
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Exhibit Grade5.4. Final Model - Behavioral Variables, Demographic Variables, Other Variables, MEPA 
Levels, and MCAS 


Variable 

Estimate 

S.E. 

Pr > 1 1 1 

Behavioral variables 




Attendance rate, end of year 

-4.78 

0.31 

<.0001 

Suspensions, end of year 

0.09 

0.03 

.007 

Mobility - Changed schools during school year 

0.57 

0.06 

<.0001 

Demographic variables 




Low income household- Free lunch 

0.57 

0.03 

<.0001 

Low income household- Reduced price lunch 

Special Education 

0.31 

0.05 

<.0001 

Low level of need (less than 2 hours) 

0.63 

0.06 

<.0001 

Low level of need (2 or more hours) 

0.95 

0.06 

<.0001 

Moderate level of need 

1.26 

0.05 

<.0001 

High level of need 

1.81 

0.13 

<.0001 

Retained 

0.82 

0.20 

<.0001 

Sex: Female 

-0.02 

0.02 

.451 

Urban residence 

0.11 

0.04 

0.003 

Overage for grade 

0.40 

0.04 

<.0001 

Other variables 




School wide Title 1 

0.38 

0.04 

0.001 

Target Title 1 

0.95 

0.05 

<.0001 

MEPA Levels 




Low level (Beginner to intermediate) 

4 th Grade MCAS Proficiency Levels 

2.21 

0.13 

0.0007 

MATH 




Warning 

4.37 

0.09 

<.0001 

Needs Improvement 

2.46 

0.04 

<.0001 

Proficient 

0.95 

0.04 

<.0001 

ENGLISH 




Warning 

1.60 

0.08 

<.0001 

Needs Improvement 

0.64 

0.05 

<.0001 

Proficient 

0.28 

0.05 

<.0001 

R A 2=0.492 

Number of observations=67403 




Fifth Grade: Illustration of Levels of Risk and MCAS Outcomes Using the Final Model 


The following box plots show the distribution of test scores by increased risk in ELA and math, 
respectively, using our Final model. As the risk level increases (x axis), the proficiency level appears to 

decrease (y axis). 





46 | Page 


G6 MCAS Math G6 MCAS ELA 


Exhibit Grade5.5 Final Model- Box plot distribution. Grade 5 Risk and 6th Grade ELA outcome 
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it Grade5.6 Final Model- Box plot distribution. Grade 5 and 6th Grade Math outcome 
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Based on the distributions of test scores by increased risk in ELA and math, respectively, the levels of risk 
are defined as follows: 

• Low Risk (approximately 75% or more of students meet the outcome variable): Intervals 1-3; 

• Moderate Risk (approximately half or more than half of the students meet the outcome 
variable): Intervals 4-6; and 

• High Risk (approximately a third or less of the students meet the outcome variable): Intervals 7- 
9. 

The statistics for the Final model's three levels of risk (low risk; moderate risk and high risk) are shown in 
Exhibits Grade5.7 and Grade5.8. In summary, approximately 89 percent of students who fall into the 
low risk category have met or exceeded the proficiency level in both ELA and math on the grade 6 
MCAS. Of the students who are categorized in the moderate risk category, 52 percent of the students 
have met the outcome. Among the high risk students approximately 15 percent met the outcome 
variable and 85 percent of the students scored below proficient on the six grade ELA and/or math MCAS. 
Note that as assessment data (MCAS test scores) were included in the risk model, the prediction 
accuracy rates in the late elementary group analysis increased, and the actual numbers of students that 
were incorrectly identified (either students classified as 'low risk' who actually did not meet the 
threshold or students classified as 'high risk' who actually met or exceeded the proficiency threshold) 
decreased in comparison with the numbers from the early elementary age group analyses. 


Exhibit Grade5.7. Final Model - Risk Level Based on Box Plot Distributions, Grade 5 


Total numbers of students in sam 

pie by risk levels 

Increased risk 

level 

Estimate For 
Probability of 

Risk 

Frequency 

Percent 

No to low 

risk 

Moderate risk 

High risk 

1 

<0.1 

15,227 

22.6 

15,227 

0 

0 

2 

>0.1 &< 0.2 

9,513 

14.1 

9,513 

0 

0 

3 

>0.2 &< 0.3 

4,635 

6.9 

4,635 

0 

0 

4 

>0.3& < 0.4 

5,290 

7.8 

0 

5,290 

0 

5 

>0.4 & < 0.5 

2,810 

4.2 

0 

2,810 

0 

6 

>0.5 &< 0.6 

5,570 

8.3 

0 

5,570 

0 

7 

>0.6 &< 0.7 

3,497 

5.2 

0 

0 

3,497 

8 

>0.7 &< 0.8 

4,664 

6.9 

0 

0 

4,664 

9 

>0.8 

16,197 

24.0 

0 

0 

16,197 

Total 


67,403 

100.0 

29,375 

13,670 

24,358 
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Exhibit Grade5.8. Final Model - Predictive Probability of Proficiency Based on Risk Level, Grade 5 
Predictive Probability of Proficiency Based on Risk Level 



Proficiency 



Below 

Proficient 


Risk Level 

Threshold 

or Above 

Total 

Low 

3,127 

26,248 

29,375 

10 . 65 % 

89 . 35 % 


Moderate 

6,562 

48 . 00 % 

7,108 

52 . 00 % 

13,670 

High 

20,717 

3,641 

24,358 

85 . 05 % 

14 . 95 % 



30,406 

36,997 

67,403 

Total 

56 . 89 % 

44 . 11 % 



Missing MCAS Recode: 

As mentioned earlier, ESE determined that students with missing data in pretest measures were 
automatically identified as 'moderate risk'. Although we do not provide a table, the descriptive statistics 
show a tendency that students without pretest measures have a higher percentage on some key 
variables in comparison with students having MCAS pretest measures - school mobility (28.2% vs. 2. 5%), 
free lunch (41.9% vs. 24. 7%), overage for grade (19.3% vs. 12.1%), urban (47.1% vs. 29. 5%), immigration 
(18.3% vs. 1.2%), MEAP level 1(8.1% vs.0.1%), MEAP level 2 (5.3% vs. 0.2%), MEPA level 3 (7.0% vs. 1.1%). 
The following tables show the distributions of 6 th grade MCAS proficiency in ELA and math for cases 
without pretest measures. Overall, the proportion of students with missing MCAS scores who failed to 
meet the academic goal mirror the proportion of moderate students at risk who met the outcome. 

Exhibit Grade5.9. Late Elementary Outcome Variable (6th Grade MCAS Proficiency in ELA and Math) 
for Cases without Pretest Measures, Grade 5 


Grade 6 

Outcome 

Frequency 

Percent 

Cumulative 

Frequency 

Cumulative 

Percent 

0 

1,538 

59.82 

1,538 

59.82 

1 

1,033 

40.18 

2,571 

100.00 
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Exhibit Grade5.10. 6th Grade MCAS Proficiency in ELA for Cases without Pretest Measures, Grade 5 



ELA Proficiency 

Frequency 

Percent 

Cumulative 

Frequency 

Cumulative 

Percent 

Above Proficient 

292 

11.36 

292 

11.36 

Needs 

Improvement 

663 

25.79 

955 

37.15 

Proficient 

1,175 

45.70 

2,130 

82.85 

Warning 

441 

17.15 

2,571 

100.00 


Exhibit Grade5.11. 6th Grade MCAS Proficiency in Math for Cases without Pretest Measures, Grade5 



Math Proficiency 

Frequency 

Percent 

Cumulative 

Frequency 

Cumulative 

Percent 

Above Proficient 

486 

18.90 

486 

18.90 

Needs Improvement 

708 

27.54 

1,194 

46.44 

Proficient 

688 

26.76 

1,882 

73.20 

Warning 

689 

26.80 

2,571 

100.00 


50 | Page 






Sixth Grade: Analysis Results and Predicted Risk Levels 

For sixth grade, several models were tested to: 1) identify individual indicators of risk and 2) identify the 
risk model that is predictive of whether a rising sixth grade student is at risk of not meeting the outcome 
variable of achieving a score that is proficient or higher in both ELA and math MCAS in grade 6(Exhibit 
Grade6.1). 


Exh bit Grade6.1 Overview of Sixth Grade Risk Indicators 


Grade: 


6 (using data from grade 5 students) 


Age Grouping: 


Late elementary (4 through 6th grade) 


Risk Indicators Tested: 


Behavioral variables 

• Suspensions, fall (not included in Final model) 

• Suspensions, end of year 

• Attendance rate, fall (not included in Final model) 

• Attendance rate, end of year 

• Mobility (more than one school within the school year) 

• Retention 
Demographic variables 

• Low income household- Free lunch 

• Low income household- Reduced price lunch 

• Special education level variables (4 total) 

• ELL status (not included in Final model) 

• Former LEP (not included in Final model) 

• Immigration status (not included in Final model) 

• Gender 

• Urban residence 

• Overage for grade (age 11 or older by Sept 1 
Other individual student variables 

• School wide Title I 

• Targeted Title I 
MEPA levels 31 

• Beginner 

• Early Intermediate 

• Intermediate 

• Transitioning to regular classes I 

• Transitioning to regular classes II 
3rd Grade MCAS Proficiency Levels 

• Math 

Warning 

Needs Improvement 
Proficient 


st of 5 th grade year) 


o 

o 

o 

• English 

o 

o 

o 


Warning 

Needs Improvement 
Proficient 


Academic Goal/ 


Outcome Variable: 


32 


Proficient or higher on the sixth grade English language arts and Math 
MCAS; Not proficient on either is equal to zero for this outcome 


Due to small samples in MEPA categories, levels were aggregated for final analyses: Levels 1-3 (beginner to intermediate) 
were made into one category, and level 4 (Transitioning to regular classes I) was kept as a separate variable. The highest level 
and non-MEPA were used as a reference. 
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NOTE: A total of 69,452 observations included this combined outcome variable for the final model. Approximately 55 percent 
were characterized as proficient or above in both math and ELA, and the remaining 45 percent were less than proficient in one 
or both. 


Sixth Grade: Simple Logistics - Analysis of Individual Indicators 

In order to build the most efficient and accurate model for the EWIS, we first examined a number of 
behavioral, demographic, and other indicators tied to individual students that may be considered in the 
resulting risk model. This analysis relied on simple logistic regressions for each individual indicator. The 
single indicator analyses allowed us to evaluate the statistical significance and coefficient for each 
indicator (Exhibit Grade6.2). This analysis was used to inform the construction of the risk models tested. 

Exhibit Grade6.2. Simple Logistic Regression Overview, Grade 6 


Simple Logistic regression: Individual indicators (predictor) 


Variable 

Estimate 

S.E. 

Pr > ChiSq 

R-Square 

N 

Demographic variables (Yes/No) 






Low income household- Free lunch 

1.58 

0.02 

<.0001 

0.1068 

66,594 

Low income household- Reduced 
price lunch 

0.99 

0.03 

<.0001 

Special education 






Low level of need (less than 2 

hours) 

1.22 

0.05 

<.0001 

0.1566 

66,594 

Low level of need (2 or more 

hours) 

1.90 

0.05 

<.0001 

Moderate level of need 

2.45 

0.04 

<.0001 

High level of need 

3.79 

0.11 

<.0001 

Immigration statust 

0.22 

0.08 

0.0043 

0.0001 

66,594 

Sex: Female 

-0.21 

0.02 

<.0001 

0.0027 

66,594 

ELL status 

1.61 

0.05 

<.0001 

0.0188 

66,594 

Former LEP studentt 

0.36 

0.05 

<.0001 

0.0009 

66,594 

Overage for grade 

1.35 

0.03 

<.0001 

0.0442 

66,594 

Urban residence 

1.07 

0.02 

<.0001 

0.0559 

66,594 

Suspension 






Suspensions, fallt 

0.49 

0.11 

<.0001 

0.0004 

66,354 

Suspensions, end of year 

0.87 

0.04 

<.0001 

0.0144 

66,506 

Attendance 






Attendance rate, fallt 

-4.44 

0.17 

<.0001 

0.0110 

66,354 

Attendance rate, end of year 

-11.26 

0.24 

<.0001 

0. 0366 

66,506 

Mobility - Changed schools during 
school year (Yes/No) 

1.21 

0.06 

<.0001 

0.0075 

66,594 

Retained (Yes/No) 

1.74 

0.15 

<.0001 

0. 0026 

66,594 


For running the statistical regression models, the outcome variable was recoded to predict the risk/likelihood of not being 
proficient or higher on the sixth third grade ELA and/or Math MCAS. 
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Simple Logistic regression: Individual indicators (predictor) 


Variable 

Estimate 

S.E. 

Pr > ChiSq 

R-Square 

N 

Title 1 participation (Yes/No) 






School-wide 

1.24 

0.02 

<.0001 

0. 0786 

66,594 

Targeted 

1.55 

0.04 

<.0001 

MEPA Levels (Yes/No) 






Low level (Beginning to 

Intermediate) 

3.81 

0.22 

<.0001 

0.0310 

66,594 

Transitioning to regular classes 1 

2.17 

0.08 

<.0001 

Transitioning to regular classes II 

0.25 

0.07 

0.0008 

5 th Grade MCAS Proficiency Levels 






MATH 






Warning 

6.66 

0.08 

<.0001 

0.425 

66,594 

Needs Improvement 

3.52 

0.04 

<.0001 

Proficient 

1.40 

0.04 

<.0001 

ENGLISH 






Warning 

6.53 

0.13 

<.0001 

0.337 

66,594 

Needs Improvement 

3.58 

0.03 

<.0001 

Proficient 

1.20 

0.03 

<.0001 


Exhibit Reads: students with a high level of need are 3.79 higher in the log-odds of not being proficient in ELA and/or Math 
MCAS than others. 

tlndicator was removed from final analyses, because either the direction of the coefficient of the variable was changed, or it 
was not statistically significant at an alpha level of .05 and the estimated coefficient is nearly zero. 

Sixth Grade Overview of Final Model 

Exhibit Grade6.3 provides the summary statistics for the Final model. The estimates in column 2 denote 
the expected difference in the log-odds of not being proficient in our combined outcome variable— sixth 
grade ELA and/or Math MCAS, holding constant other variables in the model. With the exception of 
attendance and gender, all other variables are statistically positively associated with the recoded 
outcome variable. For example, students that moved schools during the school year are expected to 
score 0.43 points higher than other students in the log-odds of not being proficient in ELA and/or Math 
MCAS, holding other variables constant. 
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Exhibit Grade6.3. Final Model - Behavioral Variables, Demographic Variables, Other Variables, MEPA 
Levels, and MCAS 


Variable 

Estimate 

S.E. 

Pr > 1 1 1 

Behavioral variables 




Attendance rate, end of year 

-4.94 

0.32 

<.0001 

Suspensions, end of year 

0.09 

0.03 

0.002 

Mobility - Changed schools during school year 

0.43 

0.07 

<.0001 

Demographic variables 




Low income household- Free lunch 

0.52 

0.03 

<.0001 

Low income household- Reduced price lunch 

Special Education 

0.29 

0.05 

<.0001 

Low level of need (less than 2 hours) 

0.59 

0.07 

<.0001 

Low level of need (2 or more hours) 

0.90 

0.06 

<.0001 

Moderate level of need 

1.23 

0.05 

<.0001 

High level of need 

1.98 

0.14 

<.0001 

Retained 

0.10 

0.02 

<.0001 

Gender: Female 

-0.08 

0.02 

0.001 

Urban residence 

0.16 

0.04 

<.0001 

Overage for grade 

0.27 

0.04 

<.0001 

Other variables 




Target Title 1 

0.69 

0.04 

<.0001 

School wide Title 1 

0.19 

0.06 

<.0001 

MEPA Levels 33 




Low (beginner to intermediate) 

5 th Grade MCAS Proficiency Levels 

4.17 

0.16 

<.0001 

MATH 




Warning 

5.04 

0.08 

<.0001 

Needs Improvement 

2.98 

0.04 

<.0001 

Proficient 

1.30 

0.04 

<.0001 

ENGLISH 




Warning 

2.37 

0.14 

<.0001 

Needs Improvement 

1.61 

0.05 

<.0001 

Proficient 

0.42 

0.04 

<.0001 

R A 2=0.494 

Number of observations=69452 




33 Fifth grade introduces 5 MEPA levels. Levels 1-3 are categorized as low and Level 4 is categorized as Transitioning to regular 

classes 1. Level 5 (Transitioning to regular classes II and non- 

-MEPA are the reference category. 
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G6 MCAS Math 2 : G6 MCAS ELA 


Sixth Grade: Illustration of Levels of Risk and MCAS Outcomes Using the Final Model 

The following box plot shows the distribution of test scores by increased risk, using our Final model. As 
the risk level increases (x axis), the proficiency level appears to decrease (y axis). 


Exhibit Grade6.4. Final Model- Box plot distribution 6th Grade ELA outcome 


280 


260 


240 


220 


200 


lit Grade6.5. Final Model- Box plot distribution 6th Grade Math outcome 



123456789 

Increased risk level 



123456789 

Increased risk level 
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Based on the distributions of test scores by increased risk in ELA and math, respectively, the levels of risk 
are defined as follows: 

• Low Risk (approximately 75% or more of students meet the outcome variable) : Intervals 1-3; 

• Moderate Risk (approximately half or more than half of the students meet the outcome 
variable) : Intervals 4-6; and 

• High Risk (approximately a third or less of the students meet the outcome variable) : Intervals 7- 
9. 

The statistics for the Final model's three levels of risk (low risk; moderate risk and high risk) are shown in 
Exhibits Grade6.6 and Grade6.7. In summary, approximately 90 percent of students who fall into the 
low risk category have met the proficiency in both ELA and math on the 6 th grade MCAS. Of the students 
who are categorized in the moderate risk category, approximately 51 percent of the students have met 
the outcome. Among the high risk students approximately 12 percent met the outcome proficiency 
level and 88 percent of the students scored below proficient on the six grade proficiency in ELA or math. 


Exhibit Grade6.6. Final Model - Risk Level Based on Box Plot Distributions, Grade 6 


Total numbers of students in sam 

pie by risk levels 

Increased 

risk level 

Estimate For 
Probability of 
Risk 

Frequency 

Percent 

No to low 

risk 

Moderate 

risk 

High risk 

1 

<0.1 

16,652 

24 

16,652 

0 

0 

2 

>0.1 &< 0.2 

12,377 

17.8 

12,377 

0 

0 

3 

>0.2 &< 0.3 

3,365 

4.8 

3,365 

0 

0 

4 

>0.3& < 0.4 

2,657 

3.8 

0 

2,657 

0 

5 

>0.4 &< 0.5 

4,797 

6.9 

0 

4,797 

0 

6 

>0.5 &< 0.6 

2,765 

4 

0 

2,765 

0 

7 

>0.6 &< 0.7 

2,655 

3.8 

0 

0 

2,655 

8 

>0.7 &< 0.8 

4,015 

5.8 

0 

0 

4,015 

9 

>0.8 

20,169 

29 

0 

0 

20,169 

Total 


69,452 

100 

32,394 

10,219 

26,839 
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Exhibit Grade6.7. Final Model - Predictive Probability of Proficiency Based on Risk Level, Grade 6 


Predictive Probability of Proficiency Based on Risk Level 



Proficiency 



Below 

Proficient 


Risk Level 

Threshold 

or Above 

Total 

Low 

3,123 

29,271 

32,394 

9 . 64 % 

90 . 36 % 


Moderate 

4,958 

5,261 

10,219 

48 . 52 % 

51 . 48 % 


High 

23,576 

3,263 

26,839 

87 . 83 % 

12 . 16 % 



31,657 

37,795 

69,452 

Total 

45 . 58 % 

54 . 42 % 



57 | Page 




Late Elementary Validation: Comparison of 2008-09 to 2009-10 Cohort 

In order show the strength of the Final model in other cohorts, the following tables examine the extent 
to which the developed risk model using the original cohort data correctly identified at-risk students in 
the validation cohort among those who actually met the predefined outcome measure (proficient or 
advanced score on both ELA and Math Grade 6 MCAS). Exhibit Late Elementary Validation. 1 shows that 
overall the predictive probability of proficiency by risk level is very similar between the original cohort 
and the validation cohort in grades 4-6. 

Exhibit Late Elementary Validation. 2 shows the output from running the logistical regression of grade 4, 
5, and 6 models using the original cohort and the validation cohort. The coefficients are all generally 
similar in magnitude and significance, except for urban resident and proficient in ELA in Grade 5, where 
the coefficients declined and were no longer significant in the validation year. In addition, the directions 
of the coefficients are the same between the models. However, attention will continue to be paid to the 
magnitude of the variable in the upper grades. 

In sum, the validation work suggests that the Final model for late elementary is strong across cohorts. 
The consistency of the coefficients between cohorts implies that the selected indicators are behaving 
similarly in reference to our outcome variable in the different groups. We will continue to test the 
prediction accuracy and stability of the EWIS models for other cohorts as more recent data sets become 
available. 
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Exhibit Late Elementary Validation. 1 Predictive Probability of Proficiency Original Cohort vs. 
Validation Cohort, Grades 4-6 


Predictive Probability of Proficiency Based on Risk Level 
FOURTH GRADE 

Risk Level 

Below Threshold 

Proficient or Above 

2009-10 

cohort 

2008-09 

cohort 

2009-10 

cohort 

2008-09 

cohort 

Low 

3,375 

11.83% 

2,424 

10.58% 

25,153 

88.17% 

20,482 

89.42% 

Moderate 

5,270 

45.63% 

5,612 

41.07% 

6,279 

54.37% 

8,052 

58.93% 

High 

19,121 

84.24% 

22,282 

81.13% 

3,576 

15.76% 

5,183 

18.87% 

Total 

27,766 

44.23% 

30,318 

47.35% 

35,008 

55.77% 

33,717 

52.65% 

Predictive Probability ol 
FIF 

F Proficiency Based on Risk Level 
: TH GRADE 

Risk Level 

Below Threshold 

Proficient or Above 

2009-10 

cohort 

2008-09 

cohort 

2009-10 

cohort 

2008-09 

cohort 

Low 

3,641 

14.95% 

3,711 

13.19% 

20,717 

85.05% 

24,754 

86.96% 

Moderate 

7,108 

52.00% 

6,892 

51.12% 

6,562 

48.00% 

6,589 

48.88% 

High 

26,248 

89.35% 

21,802 

86.81% 

3,127 

10.65% 

21,082 

13.19% 

Total 

36,997 

44.23% 

31,685 

47.84 

30,406 

55.77% 

31,685 

52.16% 

Predictive Probability ol 
Sl> 

F Proficiency Based on Risk Level 
(TH GRADE 

Risk Level 

Below Threshold 

Proficient or Above 

2009-10 

cohort 

2008-09 

cohort 

2009-10 

cohort 

2008-09 

cohort 

Low 

3,123 

9.64% 

3,346 

10.91% 

29,271 

90.36% 

27,323 

89.09% 

Moderate 

4,958 

48.52% 

5,603 

52.41% 

5,261 

51.48% 

5,088 

47.59% 

High 

23,576 

87.83% 

23,957 

89.50% 

3,263 

12.16% 

2,811 

10.50% 

Total 

31,657 

45.58 

32,906 

48.30% 

37,795 

54.42% 

35,222 

51.70% 
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Exhibit Late Elementary Validation. 2. Overview of Findings by Cohort Using Final Model 



Grade 4 

Grade 5 

Grade6 

Variable 

Original 

Cohort 

(2009-10) 

Validation 

Cohort 

(2008-09) 

Original 

Cohort 

(2009-10) 

Validation 

Cohort 

(2008-09) 

Original 

Cohort 

(2009-10) 

Validation 

Cohort 

(2008-09) 

Behavioral variables 







Attendance rate, end of year 

-3.96*** 

-4 29*** 

-4.78*** 

-3 99*** 

_4 g4*** 

_5 27*** 

Suspensions, end of year 

q 22*** 

q 22*** 

0.09** 

0.18*** 

0 09*** 

q 22*** 

Mobility-Changed schools during sy 

0.16* 

0.32** 

0 57*** 

q 22*** 

0.43*** 

0.82*** 

Demographic variables 







Low income household-Free lunch 

0 52*** 

0.58*** 

0 57*** 

0.68*** 

0 52*** 

0.50*** 

Low income household-Reduced price 

0 35*** 

0.36*** 

0 32*** 

0.38*** 

0 29*** 

0.28*** 

Special Education 







Low level of need (less than 2 hours) 

0 59*** 

0.60*** 

0.63*** 

0.69*** 

0 59*** 

0.62*** 

Low level of need (2 or more hours) 

0 95*** 

2 04*** 

0.95*** 

Q gg*** 

0 90*** 

0.78*** 

Moderate level of need 

2 22* ** 

2 35* ** 

1.26*** 

1.26*** 

1 23*** 

2 23* * * 

High level of need 

2 73*** 

2 01 *** 

2 32*** 

2 25* ** 

2 gg* ** 

2 37*** 

Gender: Female 

-0.14*** 

-0.16*** 

-0.02 

-0 09*** 

-0.08*** 

-0.18*** 

Urban residence 

0.03 

0.02 

q 22* ** 

0.04 

0.16*** 

Q Q4*** 

Overage for grade 

q 42*** 

0 23*** 

Q 4Q*** 

0.50*** 

q 27* ** 

0.34*** 

Retained 

Q 4g*** 

0 89* ** 

0.82*** 

0.50*** 

0.10*** 

0.06*** 

Other variables 







Target Title 1 

0.65*** 

0.70*** 

0.95*** 

0.82*** 

0.69*** 

Q gg*** 

School wide Title 1 

0.36*** 

q 42*** 

0.38*** 

0 49*** 

0 29*** 

0.39*** 

MEPA Levels 







Low Levels 

.12* 

0.14* 

2 2i*** 

1.00* 

4 27*** 

3.26*** 

MCAS Proficiency Levels 







MATH 







Warning 

g 7g*** 

4.07*** 

4 37*** 

4 42*** 

5.04*** 

5.08*** 

Needs Improvement 

2 45*** 

2.67*** 

2.46*** 

2 25*** 

2 gg*** 

3 02*** 

Proficient 

1 24*** 

2 27* ** 

0.95*** 

0.76*** 

2 30* ** 

2 29* ** 

ENGLISH 







Warning 

2 71*** 

2.80*** 

1.60*** 

2.06*** 

2 37*** 

2 59*** 

Needs Improvement 

2 01*** 

1 97*** 

0.64*** 

^ 4Q*** 

1.61*** 

2 4g*** 

Proficient 

^ 04*** 

Q gg*** 

0.28*** 

.16 

q 42*** 

0 29*** 


* Significant at 10%, **Significant at 5%, ***Significant at 1 
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Appendix A.l 

The following serves an example of the indicators by risk level. The example in Exhibit A.l shows the 
descriptive statistics for each indicator by risk level for the Final risk model for First Grade. 

Descriptive Statistics for Indicators by Increased Risk Level (Final Model), First Grade 


Increased 

Estimate For 

Variable 

Mean 

Std Dev 

risk level 

Probability 





of Risk 




All levels 

n/a 

Grade 3 ELA MCAS Scaled Score 

244.43 

13.75 

N=59468 


Attendance rate 

0.95 

0.06 



Suspensions 

0.01 

0.19 



Mobility - Changed schools during school 

0.05 

0.21 



year 

Gender: Female 

0.49 

0.50 



Low income household-Free lunch 

0.22 

0.41 



Low income household-Reduced price 
lunch 

0.05 

0.21 



ELL Status 

0.08 

0.27 



Overage for grade 

0.06 

0.24 



Low level of need (less than 2 hours) 

0.04 

0.19 



Low level of need (2 or more hours) 

0.02 

0.14 



Moderate level of need 

0.04 

0.20 



High level of need 

0.02 

0.14 



Urban residence 

0.31 

0.46 



Target Title 1 

0.27 

0.44 



School wide Title 1 

0.02 

0.13 

1 

<0.1 

Grade 3 ELA MCAS Scaled Score 

255.00 

11.25 

N= 420 


Attendance rate 

0.97 

0.02 

students 


Suspensions 

0.00 

0.00 



Mobility - Changed schools during school 

0.00 

0.00 



year 

Gender: Female 

1.00 

0.00 



Low income household-Free lunch 

0.00 

0.00 



Low income household-Reduced price 
lunch 

0.00 

0.00 



ELL Status 

0.00 

0.00 



Overage for grade 

0.00 

0.00 



Low level of need (less than 2 hours) 

0.00 

0.00 



Low level of need (2 or more hours) 

0.00 

0.00 



Moderate level of need 

0.00 

0.00 



High level of need 

0.00 

0.00 



Urban residence 

0.00 

0.00 



Target Title 1 

0.00 

0.00 



School wide Title 1 

0.00 

0.00 

2 

>0.1 &< 0.2 

Grade 3 ELA MCAS Scaled Score 

251.48 

11.36 

N= 15,069 


Attendance rate 

0.96 

0.03 

students 


Suspensions 

0.00 

0.01 



Mobility - Changed schools during school 

0.01 

0.08 



year 

Gender: Female 

0.66 

0.47 
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Increased 

Estimate For 

Variable 

Mean 

Std Dev 

risk level 

Probability 





of Risk 






Low income household-Free lunch 

0.00 

0.01 



Low income household-Reduced price 
lunch 

0.00 

0.04 



ELL Status 

0.00 

0.06 



Overage for grade 

0.03 

0.16 



Low level of need (less than 2 hours) 

0.00 

0.04 



Low level of need (2 or more hours) 

0.00 

0.00 



Moderate level of need 

0.00 

0.00 



High level of need 

0.00 

0.00 



Urban residence 

0.05 

0.22 



Target Title 1 

0.02 

0.15 



School wide Title 1 

0.00 

0.01 

3 

>0.2 &< 0.3 

Grade 3 ELA MCAS Scaled Score 

247.35 

11.87 

N= 16,471 


Attendance rate 

0.96 

0.04 

students 


Suspensions 

0.00 

0.05 



Mobility - Changed schools during school 

0.02 

0.15 



year 

Gender: Female 

0.48 

0.50 



Low income household-Free lunch 

0.02 

0.13 



Low income household-Reduced price 
lunch 

0.02 

0.15 



ELL Status 

0.01 

0.12 



Overage for grade 

0.05 

0.21 



Low level of need (less than 2 hours) 

0.02 

0.15 



Low level of need (2 or more hours) 

0.00 

0.02 



Moderate level of need 

0.00 

0.00 



High level of need 

0.00 

0.00 



Urban residence 

0.15 

0.36 



Target Title 1 

0.10 

0.30 



School wide Title 1 

0.00 

0.06 

4 

>0.3& < 0.4 

Grade 3 ELA MCAS Scaled Score 

243.38 

12.52 

N= 8571 


Attendance rate 

0.95 

0.05 

students 


Suspensions 

0.00 

0.10 



Mobility - Changed schools during school 

0.04 

0.20 



year 

Gender: Female 

0.39 

0.49 



Low income household-Free lunch 

0.15 

0.36 



Low income household-Reduced price 
lunch 

0.08 

0.27 



ELL Status 

0.05 

0.22 



Overage for grade 

0.06 

0.23 



Low level of need (less than 2 hours) 

0.07 

0.26 



Low level of need (2 or more hours) 

0.01 

0.09 



Moderate level of need 

0.00 

0.06 



High level of need 

0.00 

0.00 



Urban residence 

0.34 

0.47 
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Increased 

Estimate For 

Variable 

Mean 

Std Dev 

risk level 

Probability 





of Risk 






Target Title 1 

0.28 

0.45 



School wide Title 1 

0.02 

0.14 

5 

>0.4 &< 0.5 

Grade 3 ELA MCAS Scaled Score 

240.41 

12.67 

N= 5526 


Attendance rate 

0.94 

0.06 

students 


Suspensions 

0.01 

0.09 



Mobility - Changed schools during school 

0.07 

0.26 



year 

Gender: Female 

0.46 

0.50 



Low income household-Free lunch 

0.47 

0.50 



Low income household-Reduced price 
lunch 

0.12 

0.33 



ELL Status 

0.09 

0.29 



Overage for grade 

0.06 

0.23 



Low level of need (less than 2 hours) 

0.09 

0.29 



Low level of need (2 or more hours) 

0.03 

0.17 



Moderate level of need 

0.03 

0.18 



High level of need 

0.00 

0.00 



Urban residence 

0.56 

0.50 



Target Title 1 

0.51 

0.50 



School wide Title 1 

0.05 

0.21 

6 

>0.5 &< 0.6 

Grade 3 ELA MCAS Scaled Score 

237.39 

12.74 

N= 5041 


Attendance rate 

0.93 

0.09 

students 


Suspensions 

0.02 

0.19 



Mobility - Changed schools during school 

0.08 

0.27 



year 

Gender: Female 

0.44 

0.50 



Low income household-Free lunch 

0.63 

0.48 



Low income household-Reduced price 
lunch 

0.10 

0.30 



ELL Status 

0.16 

0.37 



Overage for grade 

0.06 

0.25 



Low level of need (less than 2 hours) 

0.05 

0.22 



Low level of need (2 or more hours) 

0.05 

0.21 



Moderate level of need 

0.10 

0.30 



High level of need 

0.00 

0.00 



Urban residence 

0.69 

0.46 



Target Title 1 

0.64 

0.48 



School wide Title 1 

0.03 

0.18 

7 

>0.6 &< 0.7 

Grade 3 ELA MCAS Scaled Score 

234.60 

13.09 

N=3842 


Attendance rate 

0.92 

0.09 

students 


Suspensions 

0.03 

0.23 



Mobility - Changed schools during school 

0.11 

0.31 



year 

Gender: Female 

0.36 

0.48 



Low income household-Free lunch 

0.68 

0.47 



Low income household-Reduced price 

0.07 

0.26 


lunch 
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Increased 

Estimate For 

Variable 

Mean 

Std Dev 

risk level 

Probability 





of Risk 






ELL Status 

0.29 

0.46 



Overage for grade 

0.10 

0.30 



Low level of need (less than 2 hours) 

0.05 

0.21 



Low level of need (2 or more hours) 

0.05 

0.21 



Moderate level of need 

0.17 

0.37 



High level of need 

0.01 

0.08 



Urban residence 

0.73 

0.45 



Target Title 1 

0.67 

0.47 



School wide Title 1 

0.04 

0.19 

8 

>0.7 &< 0.8 

Grade 3 ELA MCAS Scaled Score 

231.59 

12.88 

N= 2381 


Attendance rate 

0.92 

0.10 

students 


Suspensions 

0.04 

0.32 



Mobility - Changed schools during school 

0.12 

0.32 



year 

Gender: Female 

0.29 

0.46 



Low income household-Free lunch 

0.69 

0.46 



Low income household-Reduced price 
lunch 

0.06 

0.23 



ELL Status 

0.41 

0.49 



Overage for grade 

0.15 

0.36 



Low level of need (less than 2 hours) 

0.05 

0.22 



Low level of need (2 or more hours) 

0.09 

0.28 



Moderate level of need 

0.21 

0.41 



High level of need 

0.09 

0.28 



Urban residence 

0.69 

0.46 



Target Title 1 

0.62 

0.48 



School wide Title 1 

0.05 

0.22 

9 

>0.8 

Grade 3 ELA MCAS Scaled Score 

227.49 

12.56 

N= 2147 


Attendance rate 

0.91 

0.10 

students 


Suspensions 

0.13 

0.76 



Mobility - Changed schools during school 

0.17 

0.37 



year 

Gender: Female 

0.23 

0.42 



Low income household-Free lunch 

0.69 

0.46 



Low income household-Reduced price 
lunch 

0.07 

0.26 



ELL Status 

0.24 

0.43 



Overage for grade 

0.25 

0.43 



Low level of need (less than 2 hours) 

0.04 

0.18 



Low level of need (2 or more hours) 

0.13 

0.33 



Moderate level of need 

0.26 

0.44 



High level of need 

0.46 

0.50 



Urban residence 

0.69 

0.46 



Target Title 1 

0.60 

0.49 



School wide Title 1 

0.06 

0.24 
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